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CHAPTER  I 


THE  PURPOSE  AND  JUSTIFICATION  OF  THE  STUDY 
Objectives  of  the  Study 

Introduction* — The  writer  has  been  interested  for 
a number  of  years  in  the  procedures  employers  use  in  hir- 
ing, firing,  and  promoting  employees*  This  interest 
was  further  stimulated  in  the  administration  of  a social 
agency  devoted  to  educational  and  vocational  guidance 
and  placement  activities  offered  on  a no  fee  basis  to 
both  employer  and  employee* 

In  speaking  to  large  numbers  of  employers  at  their 
place  of  business,  at  the  agency  office,  and  over  the 
telephone,  it  soon  became  apparent  that  the  majority 
of  employers  and  personnel  managers  were  not  clear  in 
their  terminology  as  to  what  personality  data  they  re- 
quired* Employers,  however.  Invariably  appeared  more 
interested  in  the  '‘good”  personality  than  they  were  in 
specific  job  techniques.  Employers,  personnel  managers, 
and  counselors  on  the  basis  of  past  experience  were  con- 
fident that  work  knowledge  and  experience  were  not  suf- 
ficient for  success  or  Job  satisfaction*  Personality 
traits  had  a priority  over  the  curriculum  studied  or 
the  number  of  "A”  grades  received  by  the  potential  em- 
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ployee • This  topic  has  been  excellently  portrayed  in 

i/ 

a book  by  Falk. 

Most  employers,  moreover,  were  using  some  form  of 
rating  scale  either  written  or  oral  in  evaluating  their 
new  and  old  employees.  Traits  deemed  important  by  the 
employer  or  employment  interviewer  were  often  Judged  in 
a casual  sort  of  way.  Generalities  and  bias  more  often 
than  not  were  Interwoven  in  the  decisions  made.  Impres- 
sions and  trait  data  in  the  main  were  not  recorded  syste- 
matically or  uniformly.  In  spite  of  many  obvious  limi- 
tations, the  vast  majority  of  employers  were  emphatic 
in  their  affirmation  of  the  value  of  rating  scales. 

Purpose  of  the  Vocational  Service  Rating  Scale. — 

In  this  study  an  attempt  is  made  to  construct  a reliable 
and  valid  rating  scale  with  suggested  uses  for  guidance, 
placement,  and  personality  improvement.  Efforts  will 
be  made  to  attain  the  advantages  of  previous  research 
on  rating  scales  and  at  the  same  time  eliminate  or  mod- 
ify many  of  their  technical  weaknesses.  Great  emphasis 
is  to  be  placed  on  the  importance  of  adequate  training 
of  raters.  A concise  and  constructive  manual  accompan- 
ies the  Vocational  Service  Rating  Scale  for  the  instruc- 
tion of  Judges. 

v R.  E.  Falk,  Your  High  School  Record.  Does  It  Count? 
South  Dakota  Press.  Pierre,  South  Dakota,  1944,  124  pp. 
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Justification  of  the  Study 


Importance  of  personality. — Before  delving  further 
into  the  field  of  ratings,  one  must  have  a frame  of  re- 
ference for  the  study  of  personality.  Among  the  numer- 
ous factors  which  have  led  to  studies  of  personality 
have  been  (1)  a growing  appreciation  of  the  importance 
of  personality,  (2)  interest  in  the  meaning  and  struc- 
ture of  personality,  (3)  vocational  guidance  and  voca- 
tional selection.  Personality  is  important  in  all  fields 
of  human  endeavor.  The  primary  reasons  for  the  differ- 
ence in  the  success  of  persons  of  similar  ability  and 
opportunity  is  often  found  in  the  personalities  of  the 
individuals.  Instances  of  persons  with  high  intelligence 
who  have  made  mediocre  success  in  life  may  perhaps  be 
explained  by  their  inadequate  personalities. 

A study  of  10,000  men  made  under  the  auspices  of 

if 

the  Carnegie  Foundation  showed  that  technical  training 
accounts  for  15  per  cent  of  one's  success  and  that  per- 
sonal characteristics  are  responsible  for  the  remaining 
85  per  cent.  The  most  important  qualities  necessary 
for  success  which  the  Foundation  lists  are  attitude, 
initiative,  thoroughness,  observation,  concentration, 
creative  imagination,  decision,  adaptability,  leadership, 
organizing  ability,  expression,  and  knowledge.  The  de- 

TTdT  Carlson,  How  to  Develop  Personal  Power.  Harper 
& Bros • New  York,  1937,  pp.  60  - 62. 
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velopment  of  these  traits  Is  much  more  Important  than 
the  perfection  of  trade  knowledge. 

Brewer  mentions  that  in  a study  of  4,375  men  who 
were  discharged  from  industrial  concerns,  34.2  per  cent 
were  fired  because  of  a lack  of  sufficient  skill  or  tech- 
nical training,  62.4  per  cent  were  discharged  because 
of  some  defect  in  character  or  personality,  and  3*4  per 
cent  were  let  go  for  miscellaneous  reasons.  Incompetence 
was  the  greatest  single  factor  and  accounted  for  25*3 
per  cent  of  the  discharges  with  insubordination,  gener- 
al unreliability,  and  absenteeism  following  respective- 
ly in  the  order  of  11.1  per  cent,  10.4  per  cent,  and 

0.1  per  cent. 

2/ 

Hunt  studied  4,000  discharges  in  76  business  in- 
stitutions. He  unearthed  the  following  causes  for  dis- 
missal: carelessness  14  per  cent,  non-cooperation  10 

per  cent,  laziness  10,  dishonesty  8,  attention  to  mat- 
ters outside  of  work  8,  lack  of  initiative  7,  tardiness 
7,  lack  of  effort  7,  disloyalty  3,  discourtesy  2,  and 
miscellaneous  24  per  cent.  The  over-all  findings  of 
the  study  showed  that  failure  was  due  to  two  factors, 
lack  of  specific  ability  and  poor  character  traits. 

l/  J.  M.  Brewer,  "Religion  and  Vocational  Success" , Re- 
ligious Education:  1930:  23:  39. 

2/  H.  C.  Hunt,  "Why  They  Couldn't  Hold  Their  Jobs",  Per- 
sonnel Journal:  1935:  14:  227. 
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Of  those  discharged,  only  10.1  per  cent  lacked  the  neces- 
sary skills  while  89.9  per  cent  lacked  the  necessary 
personal  qualifications. 

As  indicated  by  these  and  other  studies  personality 
traits  are  highly  important.  Constant  effort  needs  to 
be  made  in  order  to  devise  methods  leading  to  improve- 
ment of  personality  traits.  Chapter  VII  includes  cer- 
tain suggested  methods  of  personality  improvement  through 
the  use  of  the  Vocational  Service  Rating  Scale. 

Preparation  of  youth  for  .lob  adjustment.--  One  is 
impressed  with  the  fact  that  with  the  growth  of  vocation- 
al guidance  in  our  schools,  administrators  and  teachers 
have  become  more  and  more  conscious  of  the  personalities 
of  youth.  This  trend  is  now  evidenced  in  the  schools 
that  give  grades  in  personality  traits  and  in  the  many 
schools  that  Include  them  on  the  permanent  record.  Far 
too  many  teachers,  nevertheless,  still  rate  a pupil’s 
school  performance  entirely  on  the  amount  of  subject 
matter  he  acquires  and  how  thoroughly  he  masters  it. 

It  is  frequently  overlooked  or  not  entirely  recognized 
that  success  in  school  is  dependent  to  a great  extent 
on  school  habits.  The  attitudes  developed  by  the  pupil 
in  mastering  the  curriculum  are  often  the  ones  he  will 
use  in  making  adjustments  after  leaving  school;  there- 
fore, it  is  advisable  that  proper  ones  be  formed  early. 
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As  an  aid  In  determining  whether  desirable  traits  are 
being  developed,  the  rating  scale  may  be  of  great  sig- 
nificance • 

Youth  must  be  prepared  for  the  day  he  leaves  for- 
mal schooling  and  steps  out  into  the  world  of  work. 
Teachers  can  tell  more  or  less  accurately  on  the  basis 
of  past  scholastic  records  what  to  expect  from  various 
pupils  in  mathematics,  history,  and  foreign  languages. 

In  diagnosing  the  individual  along  vocational  lines  or 
in  maladjustment  in  school,  achievement  and  intelligence 
tests  and  grades  are  not  sufficient.  Shoobs  and  Gold- 

1/ 

berg  in  their  book.  Corrective  Treatment  for  Unadjusted 
Children,  indicate  the  importance  of  personality  train- 
ing. 

In  short,  personality  training  becomes  the 
shibboleth  to  test  all  pupil  and  teacher  activi- 
ty; and  it  will  not  be  otherwise  even  with  the  es- 
tablishment of  a guidance  clinic  in  each  school. 

The  total  personality  of  the  individual  must  be 

considered.  Through  the  application  of  the  Vocational 

Service  Rating  Scale  a program  may  be  formulated  for 

intelligent  and  constructive  personality  and  vocational 

planning  for  each  person. 


l/  N.  E.  Shoobs  and  G.  Goldberg,  Corrective  Treatment  for 
Unadjusted  Children.  Harper  & Bros.  New  York,  1942,  p.3* 
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CHAPTER  II 


BACKGROUND  OF  RATING  SCALES 

Early  Use  of  Rating  Scales 
Early  attempts  at  measuring  personality* — Rating 
of  people  has  always  been  a conscious  or  almost  uncon- 
scious mental  process  of  social  relationships.  From 
the  earliest  times  there  are  traces  of  the  attempts  to 
measure  personality.  The  Egyptians,  Greeks,  and  Romans 
all  were  interested  in  virtues.  Later  people  were  clas- 
sified according  to  significant  acts  and  physical  types. 
The  total  personality  impression  was  a spontaneous  judg- 
ment for  or  against  the  individual.  In  Dante's  Divine 
Comedy . which  exemplifies  the  knowledge  of  the  Middle 

Ages,  a widely  known  example  of  character  ratincr  is  des- 

1/ 

cribed.  The  Dublin  Evening  Post  of  January  25,  1784, 
published  a rating  scale  for  estimating  the  parliamentary 
merit  of  legislators.  This  scale  analyzed  parliamen- 
tary merit  into  nine  separate  traits.  It  did  not  use 
the  concept  of  frequency,  and  an  arbitrary  quantitative 
value  was  assigned  to  each  trait. 

1/  F.  F.  Bradshaw,  "Rating  Scales  as  of  1930" , In  Mea- 
surement and  Guidance  of  College  Students.  American 
Council  on  Education.  Wilkins  Co.  Baltimore,  Maryland, 
1933,  pp.  106-110. 
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It  is  only  in  comparatively  recent  times,  however, 
that  there  has  been  real  progress  in  this  still  frontier 
field.  The  rating  scale  is  a technique  through  which 
one  person  may  evaluate  another  without  the  participa- 
tion of  the  ratee . 

1/ 

According  to  Walters,  one  of  the  first  rating  de- 
vices used  in  industry  was  the  system  of  " character  books 
instituted  by  Robert  Owen  in  his  New  Lanark  Cotton  Mills 
in  Scotland.  Each  employee  was  provided  with  a "char- 
acter block"  on  his  bench.  Each  face  of  the  block  was 
colored  differently  and  represented  an  evaluation  of 
the  employee  from  bad  to  excellent.  Daily  reports  rep- 
resented by  the  blocks  were  recorded  in  the  character 
books  which  were  consulted  as  a basis  for  promotion. 

In  a study  of  the  inheritance  of  eminence  in  1869, 
Galton  was  the  first  person  to  have  inaugurated  the  math- 
ematical or  scientific  method  in  the  description  and 

measurement  of  traits.  In  1883  he  published  the  first 

2/ 

modem  rating  scale  on  this  subject. 

Scales  from  1900  to  1920.--  In  1906  Pearson's  scale 

which  called  for  Judgment  at  seven  different  degrees  of 

l/  J.  E.  Walters , Applied  Personnel  Administration. 

John  Wiley  & Son,  Inc.  New  York,  1931,  p.4-3. 

2/  F.  Galton,  Inquiry  into  Human  Faculty  and  Its  Devel- 
opment . E.  P.  Dutton  & Go.  New  York,  1911,  p.57* 
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intelligence  was  developed.  The  idea  of  indicating 

ratings  by  a check  mark  on  the  line  is  attributed  to 

2/ 

J.  B.  Miner. 

The  Army  Rating  Scale  of  May,  1917  and  the  Scott 
Graphic  Rating  Scale  brought  rating  scales  into  promi- 
nence. The  Army  Rating  Scale  employs  a man  to  man  com- 
parison. The  rater  places  five  individuals  on  a scale 
from  the  best  at  the  top  to  the  poorest  at  the  bottom. 
One  person  is  at  the  middle,  one  person  between  the  top 
and  the  middle,  and  another  person  between  the  bottom 
and  the  middle.  The  others  are  then  compared  with  those 
used  to  make  the  scale. 


2/ 

The  Scott  Graphic  Rating  Scale  arranges  for  a scale 
of  values  after  each  trait.  At  various  points  on  this 
scale  are  descriptions  of  the  level  of  the  trait  being 
rated. 


it/ 

Lord  and  Taylor,  a New  York  department  store,  was 

probably  the  first  large  business  organization  to  base 

1/  K.  Pearson,  H0n  the  Relationship  of  Intelligence  to 
Size  and  Shape  of  Head  and  to  Other  Physical  and  Mental 
CharacteristicsM , Biometrika:  1906  - 7?  5?  105  - 146. 

2/  F.  F.  Bradshaw,  op.  cit.,  p.  107* 

2/  W*  D.  Scott,  "The  Rating  Scale",  Psychological  Bulle- 
tin:  1918:  15:  203  - 206. 


4/  G.  D.  Halsey,  Making  and  Using  Industrial  Service 
Ratings . Harper  & Bros.  New  York,  1944,  171  pp. 
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its  program  of  training  and  general  personnel  develop- 
ment on  regular  and  systematic  ratings  of  employees. 

The  plan  was  started  in  the  fall  of  1916. 

, 1/ 

Armour  and  Company's  first  experience  in  rating 
by  a definite  systematic  method  dates  to  the  year  1919. 

At  this  time,  this  company  uses  a graphic  rating  scale 
in  their  personnel  work. 

Types  of  Rating  Scales  Used  Today 

Ranking  and  scoring  methods. — After  the  year  1920 
other  rating  techniques  were  tried  out  rather  extensive- 
ly. Today  there  are  two  general  classes  into  which  ra- 
ting scales  may  be  grouped,  namely,  ranking  and  scoring 
scales.  Ranking  scales  usually  ask  for  a grading  of 
people  or  traits  in  some  specified  oraer  usually  from 
best  to  worst.  The  merits  of  this  type  of  scale  are  its 
concreteness  and  definiteness  and  its  avoidance  of  imme- 
diate emotional  reactions.  Its  weakness  lies  in  the 
difficulty  of  placing  any  group  of  people  in  rank  order 
in  any  type  of  situation.  Scoring  scales  require  the  use 
of  Judgments  which  place  an  individual  or  a trait  on  a 
scale  of  already  defined  steps.  Discussion  of  well-known 
types  of  scoring  scales  follows. 

1 / H.  G.  Ellerd,  Rating  Supervisors.  Production  Execu- 
tive Series  No.  42,  American  Management  Association. 

New  York,  1926,  p.  3. 
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Paired- comparison  method.—  In  the  method  of  paired- 
comparison  each  person  is  compared  with  each  of  the  oth- 
er persons  who  are  being  rated.  One  trait  is  consider- 
ed at  a time.  Each  judgment  is  made  on  the  basis  of 
whether  one  person  or  the  other  is  better  on  the  partic- 
ular trait.  This  method  was  popularized  by  Thurstone 
in  his  study  of  social  values  and  is  regularly  used  by 
the  National  Institute  of  Industrial  Psychology  in  their 
vocational  guidance  program.  The  amount  of  agreement 
or  disagreement  among  the  different  judges  gives  an  in- 
dication of  the  presence  or  absence  of  the  quality  in 
the  particular  individual.  The  advantage  of  this  form 
is  that  it  gives  the  subject  an  opportunity  to  form  an 
estimate  of  his  own  characteristics  and  to  contrast  his 
judgment  with  others.  The  technique  is  very  difficult 
to  score  and  is  not  used  to  any  extent  in  industry. 

The  questionnaire  method.—  This  method  has  been 
used  from  the  earliest  times  but  was  systematized  during 
World  War  I by  Woodworth  with  his  famous  PD  scale.  The 
questionnaire  usually  takes  one  of  three  forms,  true- 
false,  yes-no,  or  some  mid-point  answer.  In  each  case 
the  correct  answer  is  underlined.  Questions  can  be  list- 
ed empirically  or  by  carefully  analyzing  the  field  being 
measured  and  on  this  basis  selecting  the  questions. 

The  value  of  this  technique  depends  to  a great  extent 
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upon  the  ability  of  the  candidate  to  answer  the  questions 
correctly.  A weakness  is  the  inability  of  the  individual 
to  give  a categorical  yes  or  no.  This  method  is  used 
but  not  extensively  in  industry  today.  The  form  may 
be  completed  either  by  the  candidate  or  by  another  per- 
son who  knows  the  particular  individual. 

Multiple-reaction  scales.—  These  scales  are  fund- 
amentally a form  of  questionnaire  in  which  there  is  a 
statement  concerning  an  issue  or  situation.  A number 
of  alternative  solutions  are  presented,  one  of  which  is 
to  be  checked.  Although  this  form  of  test  is  interest- 
ing to  the  subject  and  may  have  some  diagnostic  value, 
it  is  difficult  to  construct  and  confuses  those  of  lim- 
ited intelligence.  This  type  of  scale  has  rarely  been 
used  in  industry. 

Check  list  scales.—  The  check  list  is  a rating  form 
where  the  judge  checks  various  phrases  or  words  that 
apply  to  the  ratee.  The  check  list  has  not  enjoyed  any 
appreciable  popularity  in  school  or  industry.  The  best 

1/ 

known  check  list  is  that  of  the  Probst  Service  Report 

where  the  rater  can  check  off  any  number  of  items  up 

to  100.  It  is  recommended  that  three  judges  rate  each 

individual,  and  the  same  rating  sheet  is  used  by  the 

l/  J.  B.  Probst,  Measuring  and  Rating  Employee  Value. 
Ronald  Press.  New  York,  1947,  166  pp. 
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different  judges. 

Point  scales. — The  scales  consist  of  a number  of 
items  to  each  of  which  there  is  given  an  arbitrary  value. 
One  of  the  earliest  forms  of  this  type  of  scale  was  the 
man  to  man  scale,  originally  developed  at  the  Carnegie 
Institute  of  Technology  by  Scott  for  the  purpose  of  se- 
lecting salesmen.  It  was  later  adapted  for  the  Army 
during  World  War  I for  the  purpose  of  selecting  officers. 
The  qualities  regarded  as  necessary  for  success  are  de- 
fined, and  each  quality  is  given  a range  of  values.  The 
total  of  an  individual's  score  on  the  qualities  helps 
to  decide  his  worth  or  rank. 

Linear  scales.—  The  range  of  ability  is  indicated 
by  a straight  line  without  gradations.  One  end  of  the 
line  represents  the  least  amount  of  the  trait  and  the 
other  the  greatest.  The  judge  places  a check  mark  on 
the  line  at  the  point  which  he  believes  corresponds  with 
the  amount  of  the  trait  possessed  by  the  ratee.  This 
type  of  scale  has,  for  the  most  part,  been  replaced  by 
the  graphic  scale. 

Frequency  scales.—  In  these  scales  the  trait  is 
regarded  as  being  distributed  in  the  normal  population 
according  to  the  frequency  of  the  normal  distribution 
curve.  Each  trait  is,  therefore,  divided  on  this  basis 
into  five  to  ten  units.  Equal  intervals  on  the  scale 
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represent  equal  steps  in  frequency  on  the  normal  curve* 

The  subjects  to  be  rated  are  then  assigned  positions 
most  descriptive  of  their  behavior*  Degrees  on  this 
scale  are  expressed  in  terms  of  per  cent  or  point  values. 
Explanation  is  usually  given  regarding  the  normal  expec- 
tancy of  distribution.  For  a five  unit  scale  approxi- 
mately seven  per  cent  fall  in  each  of  the  extreme  groups, 
24  per  cent  in  each  of  the  above  and  below  average  groups, 

1/ 

and  38  per  cent  in  the  average  group*  The  weaknesses 
of  this  type  of  scale  are  often  the  use  of  abstract  terms 
and  the  analysis  of  a capacity  into  hypothetical  degrees 
which  vary  from  analyst  to  analyst* 

Graphic  rating  scales*—  In  the  graphic  rating  scale 
descriptions  in  degrees  of  a trait  are  given  at  several 
points  on  a five  or  six  inch  line.  The  judge  checks 
that  point  on  the  continuum  which  most  closely  corresponds 
to  his  interpretation  of  the  degree  to  which  the  ratee 
possesses  the  trait.  Variations  Include  weighting  the 
scale  and  using  a frequency  system  in  order  to  translate 
descriptive  content  into  quantitative  data.  The  graphic 
rating  scale  is  the  most  popular  method  and  depends  to 
a great  extent  upon  the  training  of  the  rater. 

Weighted  scoring  scales.—  In  the  case  of  weighted 

1/  H.  Moore,  Psychology  for  Business  and  Industry.  McGraw 
Hill.  New  York,  1939,  p.  220. 
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scoring  scales,  each  item  is  given  a definite  numerical 
value  based  on  an  arbitrarily  chosen  unit  or  on  a sta- 
tistically weighted  value.  The  total  score  in  each  case 
is  the  sum  of  the  values  belonging  to  the  checked  items. 

Variations  in  the  Concept  of  Personality 
Difficulty  in  classifying  concents. — Psychologists 
have  encountered  considerable  difficulty  in  classifying 
their  concepts  of  personality.  Each  psychologist  may 
define  personality  according  to  his  training,  experience 
and  bias.  There  is,  however,  no  need  for  alarm  or  con- 
sternation because  even  in  the  so-called  "exact  sciences 
such  as  physics  and  chemistry,  there  are  different  ways 
of  conceiving  basic  laws. 

Procedures  based  on  a biophysical  interpretation 
of  personality  have  assumed  that  a person  has  a single 
personality  and  that  all  raters  must  approximate  it. 

Some  investigators  assume  that  they  will  get  the  true 
personality  by  averaging  these  Judgments.  This  point  of 

' 1/ 

view  is  given  by  Allport. 

The  usual  assumption  is  that  the  closer  the 
agreement  (that  is,  the  less  the  variation)  of  the 
Judges  upon  a given  trait  the  greater  the  probabil- 
ity that  the  average  of  their  ratings  is  an  accur- 
ate measure  of  that  trait.  Variation  is  the  neces- 
sary result  of  personal  bias  away  from  the  truth; 
hence  uniformity  means  that  the  truth  is  obtained. 

1/  F.  H.  Allport,  Social  Psychology.  Houghton  Mifflin. 
Boston,  1924,  pp.  127-128. 
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In  most  methods  of  personality  measurement,  the 
underlying  hypothesis  has  been  that  personality  is  divid- 
ed into  traits  such  as  honesty,  accuracy,  cooperation, 
intelligence,  and  the  like.  Even  though  traits  may  not 
exist  as  separate  entities,  the  description  of  a person- 
ality in  terms  of  traits  is  of  value.  Naturally  no  one 
trait  functions  by  itself,  but  to  look  at  it  as  if  it 

1/ 

did  can  help  in  the  diagnosis  of  an  Individual.  Lee 
favors  this  type  of  thinking# 

Rating  scales  provide  a means  of  increasing 
the  definiteness  of  teachers*  judgments  on  traits 
which  can  not  be  measured  by  objective  means,  such 
as  Industry,  initiative,  reliability,  cooperation, 
and  leadership. 

2/ 

This  idea  is  further  substantiated  by  Lynch# 

The  only  way  to  achieve  in  the  field  of  per- 
sonality measurement  the  refinement  and  exactness 
characteristic  of  measurement  in  the  natural  sci- 
ences is  to  reduce  the  great  variety  of  human  ac- 
tivities to  a common  denominator,  objectively  ob- 
servable responses.  With  this  conception  the  prac- 
tices of  the  rating  scale  proponents  certainly  ap- 
pear to  be  in  profound  agreement. 

Many  writers  have  been  concerned  with  the  fact, 

for  example,  that  a person  is  neither  wholly  honest  nor 

dishonest.  He  may  be  honest  in  money  matters  with  one 

Individual  or  in  a specific  undertaking  and  dishonest 

1/  J.  M.  Lee,  A Guide  to  Measurement  in  the  Secondary 
3chool#  Appleton-Century  Co.  New  York,  1936,  p.  37* 

2/  J.  M.  Lynch,  "The  Psychology  of  the  Rating  Scale", 
Educational  Administration  and  Supervision:  1944:  30: 
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1/ 

with  another  person  or  in  another  circumstance.  Traxler 
discusses  the  situation  in  the  following  manner. 

It  is  true  that  the  immediate  environmental 
situation  does  color  the  personality  of  every  in- 
dividual appreciably.  No  one  feels  the  same  nor 
behaves  the  same  in  the  office,  on  the  golf  course, 
in  a New  Year’s  Eve  celebration,  and  in  Sunday  morn- 
ing devotional  services.  Nevertheless,  nearly  all 
individuals  display  enough  consistency  in  behavior 
under  even  widely  varying  conditions  so  that  those 
who  know  one  well  know  what  to  expect  of  him.  In 
other  words  within  limits  personality  qualities 
are  consistent  and  relatively  permanent  in  all  nor- 
mal individuals.  If  an  occasional  person  is  so 
erratic  that  his  behavior  is  utterly  unpredictable, 
that  very  erratic  quality  in  itself  is  a dependable 
basis  of  prediction,  and  one  that  will  in  all  prob- 
ability cause  that  individual  to  be  institutiona- 
lized. 

Personality  measurement.—  Personality  measurement 
is  still  in  its  infancy.  Among  the  most  common  fumblings 
has  been  the  identification  of  personality  with  traits 
that  are  assumed  to  have  an  independent  existence.  This 
has  taken  place  partly  because  of  the  fact  that  some 
traits  have  both  an  independent  existence  and  an  effect 
upon  other  persons.  Traits  that  have  an  independent 
existence  are  rightly  subjected  to  more  objective  measure- 
ments because  they  exist  in  an  individual  irrespective  of 

2 / 

other  persons  who  may  be  present.  Rugg  failed  to  do 

this  in  his  oft  quoted  study  of  rating  scales  where  he 

l/  A.  E.  Traxler,  '‘Measurement  in  the  Field  of  Education"  , 
Education:  19 46:  66:  424. 

2/  H.  0.  Rugg,  "Self-Improvement  of  Teachers  Through  Self- 
Ratings",  Elementary  School  Journal:  1920:  20:  670-684. 
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obtained  unfavorable  results.  He  compared  ratings  on 
intelligence  with  scores  made  on  tests  of  intelligence. 
The  ratings  were  measures  of  the  effect  of  the  ratee’s 
intelligence  upon  the  judge  rather  than  measures  of  the 
ratee’s  intelligence  which  has  an  independent  existence. 
The  objective  measurement  gives  the  independent  value  of 
the  trait.  Some  traits  have  no  independent  existence 
as  they  are  functions  of  the  relationship  between  two 
persons.  In  such  cases  the  only  form  of  measurement  pos- 
sible is  probably  that  in  subjective  terms,  because  the 
essence  of  the  measurement  is  the  effect  produced  on  the 
rater.  A complicating  factor  may  be  the  effort  to  dis- 
tinguish between  the  motive  and  the  effect.  The  person 
may  intend  cooperation,  but  the  effect  may  not  be  coop- 
eration. 

Factor  analysis  of  abilities  and  traits.—  The  work 
of  Spearman  may  be  considered  as  the  stimulus  for  the 
use  of  factor  analysis  in  the  study  of  abilities  and 
traits.  It  is  essentially  a search  by  statistical  meth- 
ods for  psychological  processes  which  underly  and  deter- 
mine test  performance.  The  results  depend  upon  the  var- 
iety of  items,  the  ambiguity  of  the  items,  and  the  var- 
iety of  the  persons  who  respond  to  the  items.  In  using 
the  method  of  factor  analysis,  if  less  than  300  persons 
and  less  than  40  items  are  used,  the  investigation  suf- 
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fers  from  a paucity  of  facts.  Greene  in  a discussion 
of  the  limitations  and  advantages  of  factor  analysis 
mentions  the  following. 

The  number  of  common  factors  found  in  any  bat- 
tery of  tests  will  vary  with  the  items  that  are  in- 
cluded and  the  abilities  of  the  persons  tested. 

Hence,  even  though  two  groups  of  persons  are  apprais- 
ed by  the  same  tests,  analyses  of  the  two  groups  will 
often  yield  different  factorial  loadings.... 

2/ 

In  studies  of  personality  Tiffin  found  two  factors, 

2/  y y 

Tschechtelin  four,  Bolanovich  six,  Lovell  thirteen 

6/ 


factorsj^plus  four  super-factors,  Layman  twelve,  and 


Kelley  thirteen  factors.  In  spite  of  its  promising 
leads,  the  field  of  factor  analysis  is  still  a very  con- 
troversial one.  Wide  variation  in  scores  has  been  re- 


ported from  tests  which  seem  very  similar. 

TT  E.  B.  Greene,  Measurement  of  Human  Behavior.  Odyssey 
Press.  New  York,  1941,  p.  147. 

2/  J.  Tiffin,  Industrial  Psychology.  Prentice-Hall. 

New  York,  1942,  pp.  231-261. 

2/  M.  A.  Tschechtelin,  "Factor  Analysis  on  Children's 
Personality  Ratine  Scale"  Journal  of  Psychology:  1944 s 
18:  197-200. 

4/  D.  J.  Bolanovich,  "Statistical  Analysis  of  an  Indus- 
trial Rating  Chart",  Journal  of  Applied  Psychology:  1946: 
30:  23-31. 

C.  Lovell,  "A  Study  of  the  Factor  Structure  of  Thir- 
teen Personality  Variables" , Educational  and  Psychological 
Measurement : 1945?  5s  333-340. 

6/  E.  McC.  Layman,  "An  Item  Analysis  of  the  Adjustment 
Questionnaire ",  Psychological  Bulletin:  1937s  34:  782. 

2/  T.  L.  Kelley,  Essential  Traits  of  Mental  Life.  Har- 
vard University  Press.  Cambridge,  1935,  143  pp. 
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Certain  investigators  question  the  value  of  providing 
pure  traits  for  an  instrument  designed  to  improve  per- 
sonality. This  view  is  pointed  out  clearly^in  My  Per- 
sonality Growth  Book  by  McCall  and  Herring. 

The  test  does  not  purport  to  provide  a list 
of  pure  traits.  If  such  pure  traits  existed,  and 
if  we  knew  what  they  were,  probably  it  would  still 
be  preferable  to  employ  in  a teaching  instrument 
like  this,  normal  well-known  recognizable  behavior 
manifestations.  The  items  and  sub-tests  overlap 
and  reinforce  one  another,  and  this  is  as  it  is  in 
life  and  should  be  both  in  school  and  in  a teach- 
ing instrument. 


Value  of  Rating  Scales 

Worth  of  scales.—  In  spite  of  much  convincing  evi- 
dence of  the  value  of  rating  scales  by  such  investiga- 
tors in  the  field  of  measurement  as  Moore,  McCall,  Rem- 
mers,  Guilford,  and  others,  it  is  necessary  to  furnish 
data  as  regards  their  worth.  Much  of  the  damage  came 

from  the  works  of  Rugg  on  the  Army  Rating  Scale  in  World 

2/ 

War  I and  from  certain  technical  weaknesses.  Rugg  made 
an  exhaustive  study  of  ratings  and  came  to  the  conclu- 
sion that  they  were  so  inaccurate  as  to  be  practically 

1/  W.  A.  McCall  and  J.  P.  Herring,  My  Personality  Growth 
Book.  Bureau  of  Publications.  Teachers  College,  Colum- 
bia University.  New  York,  1941,  p.  3* 

2/  H.  0.  Rugg,  M Self-Improvement  of  Teachers  Through 
Self-Rat ings* , Elementary  School  Journal:  1920:  20: 

670  - 684. 
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1/ 

worthless.  McCall  on  the  contrary  contends  that  rating 

scales  are  extremely  accurate. 

There  are  few  if  any  measurements  made  by  any 
science  that  are  more  accurate  than  ratings  on  per- 
sonality traits ... .Rugg  is  probably  right  if  we 
inquire  whether  a high  rating  for  Intelligence  means 
that  the  person  rated  really  has  high  intelligence. 
But  it  is  possible  that  how  much  intelligence  peo- 
ple think  an  individual  has  is  of  greater  moment 
than  how  much  he  really  has.  In  this  vitally  sig- 
nificant area  ratings  are  delicately  accurate. 

2/ 

Guilford  is  most  emphatic  as  to  the  value  of  rat- 
ing scales:  "Without  any  doubt  rating  scale  methods  have 
made  their  place  secure  in  industrial  practice  and  in 

the  educational  field." 

3/ 

Symonds  stresses  ratings  in  an  article  on  the 
evaluation  of  teacher* s personality:  "in  conclusion,  this 
discussion  has  stressed  the  Importance  of  ratings  as  a 
quantitative  record  of  judgments  as  the  basis  for  any 
program  of  personality  evaluation." 

4/ 

Watson  mentions  that  some  rating  scales  seem  to 

have  reliabilities  surpassing  most  existing  group  tests 

and  lists  the  examples  shown  in  Table  I in  substantiation 

l/  W.  A.  McCall,  Measurement . Macmillan  Co.  New  York, 
1939,  p.  311. 

2/  J.  P.  Guilford,  Psychometric  Methods.  McGraw  Hill. 

New  York,  1936,  p.  265. 

P*  M.  Symonds,  "Evaluation  of  Teacher  Personality", 
Teachers  College  Record:  1946:  48:  33 • 

4/  G.  B.  Watson,  "A  Supplementary  Review  of  Measures  of 
Personality  Traits",  Journal  of  Educational  Psychology: 
1927:  18:  76. 
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of  this  conclusion 


Table  1 


Selected  Coefficients  of  Reliability 
Listed  by  Watson  1/ 


Investigator 


Coefficient 
of  Reliability 


Knight  and  Cleeton 

3 hen 

Ferfey 


Barr 

Freyd 

Webb 


.40  to  .80 
.52  to  .87 
.65  to  .81 
.80  to  .90 
.62  to  .91 
.70  to  .97 


2/ 


Ellerd,  an  industrial  relations  executive,  summari- 
zes his  views  on  rating  scales  as  follows. 

Therefore,  I feel  confident  there  can  be  no  good 
purpose  by  discussing  whether  or  not  ratings  are 
necessary  or  desirable.  They  are  and  always  have  been 
an  integral  part  of  organization  work,  and  we  cannot 
escape  them  in  some  form  or  another. 

Rating  Scales  during  World  War  II. — During  World 
War  II  rating  scales  were  used  extensively.  Enlisted 
men  were  rated  at  each  change  of  post  and  at  each  pro- 
motion. Officers  were  evaluated  at  the  end  of  every 
30  days  consecutive  duty  at  a specific  military  occupa- 
tional specialty.  For  officers*  ratings  below  a certain 
minimum  meant  impossibility  of  promotion  regardless  of 
time  in  grade.  Ratings  were  made  by  the  commanding  of- 
1/  Ibid,  p.  76. 

2/  H.  G.  Ellerd,  Rating  Supervisors.  Production  Execu- 
tive Series  No.  42,  American  Management  Association. 

New  York,  1926,  p.  3. 
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ficer.  They  were  translated  numerically  into  one  of 
five  descriptive  terms  of  efficiency.  At  some  Officers 
Candidate  Schools,  one  of  the  elimination  devices  was 
the  method  of  having  candidates  rate  one  another  as  to 
their  chance  of  being  successful  officers. 

Graphic  rating  scale  preferred.--  Scientific  data 
concerning  the  relative  effectiveness  of  different  kinds 
of  rating  scales  are  meagre.  All  have  been  used  or 
suggested, and  each  type  has  its  followers. 

1/ 

Guthrie  in  a carefully  conducted  study  compared 
the  ranking  and  graphic  rating  methods  and  found  the 

2/ 

latter  to  be  about  15  per  cent  more  reliable.  Guilford 
favors  the  graphic  type  of  rating  scale:  "The  graphic 
type  of  rating  scale  is  by  far  the  most  popular,  and  on 
the  whole  most  satisfactory ... .There  are  no  disadvantages 
that  apply  to  the  graphic  type  of  scale  alone." 

Some  of  the  advantages  of  the  granhic  rating  scale 

if 

in  comparison  with  other  types  of  scales  listed  by  Freyd 
are : 

"it  is  simply  and  easily  grasped. 

"it  is  interesting  and  requires  little  motl- 

T7TT  R.  Guthrie,  "Measuring  Student  Opinion  of  Teachers", 
School  and  Society:  1927?  25?  175-176. 

2/  J.  P.  Guilford,  op.'clt.,  p.  8. 

2/  M.  Freyd,  "The  Graphic  Rating  Scale",  Journal  of  Edu- 
cational Psychology:  1923?  14- ? 83-102. 
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vation  of  the  rater 


"it  is  quickly  filled  out. 

"It  is  simply  and  easily  scored. 

"it  frees  the  rater  from  direct  quantitative 


terms • 

"it  enables  the  rater,  nevertheless,  to  make 
his  discrimination  as  fine  as  he  cares,  although 
the  discrimination  is  lost  if  a scoring  stencil 
of  only  a few  points  is  used. 

"The  descriptive  terms  aid  the  rater  in  that 
they  make  the  various  degrees  of  the  trait  more 
concrete .... 

"it  is  numerical;  that  is,  no  master  scale  is 
required  as  in  the  Army  Scale. 

"The  fineness  of  scoring  may  be  altered  at 
will,  yielding  scores  of  from  1 to  5 or  from  1 to 
100...."  1/ 

2/ 

Hayes  and  Paterson  report  finding  the  graphic  meth- 
od highly  reliable.  They  found  correlations  of  over 
.65  between  the  ratings  of  the  same  men  by  different 
Judges  and  of  reratings  of  the  same  men  by  the  same  judge. 

i/ 

Symonds  also  mentions  that  he  believes  that  the  graphic 
rating  scale  is  the  best  form  to  use. 

1/  Ibid. 

2/  M.  H.  S.  Hayes  and  D.  G.  Paterson,  "Experimental  De- 
velopment of  the  Graohlc  Ratincr  Method"  , Psychological 
Bulletin:  1921:  18:  98-99. 

2/  P.  M.  Symonds,  "Notes  on  Rating  versus  Ranking", 

Journal  of  Applied  Psychology:  1925?  9:  188-195. 
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Use  of  Rating  Scales  in  Industry 

Interest  expressed  by  industry .--  An  examination 
of  the  literature  revealed  insufficient  data  regarding 
the  extent  to  which  industry  employs  rating  scales.  A 
preliminary  phase  of  this  research  was  a survey  to  deter- 
mine the  use  of  industrial  rating  scales.  Two-hundred 
fifty-three  business  organizations  holding  membership 
in  the  American  Management  Association  were  contacted 
by  mail.  Each  company  was  asked  to  submit  a copy  of  its 
merit  rating  scale  and  instructions  for  its  use.  Of 
the  252  companies  which  replied,  144  or  57  per  cent  em- 
ployed rating  schedules. 

The  names  of  the  companies  will  be  omitted  unless 
information  given  the  writer  has  already  been  published. 
Many  have  preferred  to  remain  anonymous  or  marked  mater- 
ial as  confidential.  Others  have  not  specifically  grant- 
ed permission  to  publish  data. 

The  majority  of  the  concerns  were  interested  in  the 
research  and  asked  to  be  placed  on  a mailing  list  for 
any  published  results.  Only  two  cases  mentioned  speci- 
fically that  they  had  discontinued  rating  scales.  One 
personnel  manager  from  a very  large  concern  wrote  an  ex- 
tremely long  letter  to  point  out  why  his  concern  did 
not  use  rating  scales  when  this  research  had  undoubtedly 
brought  in  so  many  elaborate  rating  devices.  Realist!- 
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cally  this  manaecer  had  used  ratings  without  his  being 

1/  " 

cognizant  of  the  fact*  Taylor  summarizes  the  problem 
as  follows. 


An  employer  does  rate  all  his  employees  to 
all  intents  and  purposes,  every  time  he  promotes  one 
man  instead  of  another,  gives  one  man  a pay  increase 
instead  of  another,  or  in  any  way  changes  the  rela- 
tive status  of  various  individuals.  He  expresses 
by  these  actions  his  over-all  judgment  of  the  rela- 
tive standing  of  the  various  individuals  who  have 
received  different  treatment.  The  only  question 
which  needs  to  be  answered  is  whether  employers 
shall  rate  employees  on  a haphazard  basis  without 
recording  judgment  or  whether  they  shall  rate  em- 
ployees systematically,  regularly,  objectively,  and 
as  accurately  as  possible. 

Comparison  of  35  industrial  scales.—  A lack  of 
standardization  in  the  rating  scales  employed  in  school 
and  industry  makes  comparison  extremely  difficult  ex- 
cept in  general  terms.  For  purposes  of  describing  the 
numerous  rating  devices  used  in  industry,  35  scales 
were  selected  at  random  for  comparison.  It  is  impossible 
to  compare  each  one  of  the  35  scales  with  one  another 
in  great  detail.  Some  scales  do  not  designate  the  trait 
by  name.  Others  call  the  same  trait  by  different  names 
although  they  seem  to  be  describing  similar  behavior. 

In  examining  Table  2,  certain  major  factors  are 
brought  into  focus.  The  graphic  type  of  rating  scale  is 
used  to  the  greatest  extent.  Excluding  three  companies 
with  more  than  60  traits,  traits  run  from  20  for  Company 

v H.  C.  Taylor,  "Problems  of  Selecting  and  Evaluating 
Employees",  Purdue  University  Engineering  Bulletin  Ex- 
tension : 1938 : 23:  87* 
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Table  2 


Comparison  of  35  Rating  Scales  Used  In  Industry 


Con- 

cern 

Type  of 
Scale 

No  . of 
Traits 

Divi- 

sions 

of 

Trait 

Scor- 

ing 

Judges 

Train- 

ing 

Fre- 

quency 

of 

Rating 

1 

G 

12 

5 

X 

D 

P 

2 

G 

10 

5 

NW 

M 

P 

3 

De 

7 

4 

X 

D 

P 

4 

G 

6 

4 

NW 

D 

P 

5 

MG 

6 

8 

X 

D 

P 

6 

MCL 

106 

1 

N 

M 

P 

7 

MCL 

60 

10 

N 

D 

P 

8 

G 

8 

3-5 

X 

None 

P 

9 

G 

13 

5 

NW 

D 

P 

10 

G 

9 

5 

N 

D 

P 

11 

G 

20 

5 

NW 

D 

P 

12 

G 

14 

4 

N 

D 

P 

13 

G 

12 

5 

X 

D 

P 

14 

MCL 

12 

3-5 

X 

D 

P 

15 

G 

7 

5 

X 

D 

P 

Key: 


CL  - Check  list 
De  - Descriptive  Rating  Scale 
D - Directions 
G - Graphic  Rating  Scale 
LR  - Letter  rating 
M - Manual 

MCL  - Modified  check  list 
MG  - Modified  Graphic  Rating  Scale 
N - Numerical 
NW  - Numerically  weighted 

P - Periodic  ratings  at  least  once  a year  or  as  needed 
X - Scale  used  clinically  without  numerical  scoring 
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Table  2.  (concluded) 


Divi- 

sions 


Judges 


Fre- 

quency 


Con- 

cern 

Type  of 
Scale 

No  • of 
Traits 

of 

Trait 

Scor- 

ing 

Train- 

ing 

of 

Rating 

16 

G 

8 

10 

X 

D 

P 

17 

G 

15 

5 

N 

M 

P 

18 

MG 

10 

4 

X 

M 

P 

19 

G 

11 

5 

NW 

D 

P 

20 

De 

8 

4 

X 

D 

P 

21 

MCL 

11 

4 

N 

D 

P 

22 

G 

8 

15 

N 

D 

P 

23 

De 

8 

5 

N 

D 

P 

24 

MCL 

74 

10 

N 

D 

P 

25 

G 

11 

5 

X 

M 

P 

26 

G 

11 

4 

X 

D 

P 

27 

CL 

13 

3-5 

X 

D 

P 

28 

De 

19 

5 

N 

D 

P 

29 

G 

7 

3 

N 

D 

P 

30 

De 

10 

5 

LR 

D 

P 

31 

G 

10 

5 

N 

D 

P 

32 

G 

15 

4 

N 

D 

P 

33 

De 

9 

5 

N 

D 

P 

34 

MG 

12 

4 

N 

D 

P 

35 

G 

12 

7 

N 

D 

P 

Key: 

CL  - Check  list 
D - Directions 

De  - Descriptive  Rating  Scale 
G - Graphic  Rating  Scale 
LR  - Letter  rating 
M - Manual 

MCL  - Modified  check  list 
MG  - Modified  Graphic  Rating  Scale 
N - Numerical 
NW  - Numerically  weighted 

P - Periodic  ratings  at  least  once  a year  or  as  needed 
X - Scale  used  clinically  without  numerical  scoring 
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11  to  6 for  Companies  4 and  5.  Most  companies  use  divi- 
sions of  the  scale  ranging  from  three  to  seven.  The 
most  frequently  employed  method  of  qualification  is  the 
numerical.  Only  a few  concerns  assign  specific  weights 
to  traits.  Five  concerns  have  a manual  of  instructions* 
Instructions  comprise  those  directions  that  accompany  a 
rating  scale  and  vary  from  a few  lines  to  a page  or  two. 
Manuals  are  more  than  simple  directions,  contain  more 
information  for  raters,  and  explain  in  some  detail  the 
rating  system  and  trait  terminology.  It  did  not  seem 
advisable  to  list  the  frequency  of  rating  for  each  con- 
cern because  there  are  individual  rules  when  ratings 
are  given.  The  times  at  which  ratings  occur  may  be  changed 
because  of  special  occasions  such  as  lay-off,  promotion, 
bonuses,  and  the  like.  It  is  of  Interest,  however,  to 
note  that  Company  3 maintains  new  ratings  once  each  month 
regardless  of  any  particular  need.  Company  6 has  the 
same  procedure  every  six  months,  while  Company  20  fol- 
lows this  procedure  once  a year. 

Some  very  interesting  experiments  on  rating  scales 
have  been  conducted  by  Company  26.  At  this  concern  as 
at  some  others,  an  attempt  has  been  made  to  drop  any 
numerical  values  or  statistical  evaluation.  The  philos- 
ophy in  directing  this  action  is  that  it  is  neither  pos- 
sible nor  desirable  to  reduce  records  of  Judgment  on 


' ' 

srfT  .n9vee  od-  oertrid-  mo'tt  anlsnfi'i  slaoe  ©rid  to  cnoie 
©rid-  ei  rtol&JBQltflavp  to  boridem  Jbsxolqro©  xX^rtsuP®*1^  deoa 

roc  wat  a \LnO  •S.zot 

« 

,(  oce  • d - dd  c.  oidr  ■ t r-  a-'  *d  - *■?<  "oo  '•  cu*  d . 
,ovd  *10  95 sq  a od  e©r ril  ws"  • • " - v ' •'•  l - r;  jnids'£ 

idnoo  « enot  J09rx.f .ft  alqiaX 

EJtadeJb  .roe  nl  ni&lqxs  bn*  ,«*iadm  *xol  iieldl  •? 

• • 

-•  ..••  j • '1  ■'  ’ - rri  -■  •'  - 

■ 

• ’''OO  • ■ •.  ■ 

r ...  ■ 

, . t 

ridncra  xio 90  corro  avnldan  wait  211  ad-:.-*  ••  ?!  ' :J  'trcC  d’rid  - /on 

. Seen  nalnoidrcaq  vns  to  BaaXJbrtagen 

..  ■ * j . ; p./c  : 

Golaoe  vntf&z  no  afnerl'ieqxe  ^/tld^e'ied’nl  x?i9V  #aic 

. 

yrrc  non  ft  od  te*i©rido  ©biob  da 

-ecIJt  . 

at  dl  dertd  ’ ^ 1 d r;-/:  /.'  'd  09*1X6  nl  ydqo 

; ' +r  .0 >•  o out  : i ! o;  • : 'c.’r 


human  values  to  exact  figures.  They  do  not  believe  that 
any  total  numerical  score  is  as  indicative  of  an  indi- 
vidual’ s job  behavior  as  the  picture  revealed  by  the 
scale  Itself.  Numerical  scores,  however,  aid  in  the 
analysis  of  the  data.  Most  firms  use  some  type  of  score. 

Some  scales  attempt  to  add  all  the  traits  to  obtain 
a composite  score.  One  can  see  the  futility  and  appar- 
ent inconsistency  of  adding  together  personality  traits 
such  as  initiative,  cooperation,  reliability,  and  the 
like.  It  is  analagous  to  adding  together  oranges  and 
apples.  With  this  plan,  furthermore,  the  total  scores 
of  two  individuals  may  be  alike;  yet  one  worker  may  be 
of  greater  value  to  the  concern  for  a particular  job 
than  another.  Weakness  in  some  items  is  offset  by 
strength  in  others. 

Most  rating  scales  in  industry  have  not  followed 
the  rules  and  procedures  considered  necessary  in  construct- 
ing and  using  rating  instruments.  One  of  the  difficul- 
ties is  that  some  concerns  have  rated  performance  and 
personality  traits  on  the  same  sheet.  The  value  of  this 
procedure  is  a moot  question.  Other  concerns  include 
items  for  rating  that  can  be  objectively  measured,  e.  g. 
attendance.  Items  included  in  industrial  rating  instru- 
ments have  varied  from  concern  to  concern.  Companies 
duplicate  traits  with  different  terminology.  The  total 
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picture  of  ratings  in  industry,  nevertheless,  indicates 
much  more  agreement  than  is  evidenced  by  casual  examina- 
tion. A few  successful  industrial  rating  programs  are 
outlined  below. 

Company  A.—  Rating  scales  have  been  employed  by 
Company  A in  its  personnel  work  for  over  two  decades. 
Management  feels  they  play  an  extremely  important  part 
in  its  program.  One  of  the  scales  extensively  used  is 
a graphic  rating  instrument  for  college  graduates.  Man- 
agement believes  these  men  will  benefit  from  a thorough 
internship  before  being  transferred  to  a major  position. 
Personality  traits  and  performance  data  are  combined. 
There  are  eight  traits;  each  one  of  which  is  broken  down 
into  degrees. 

Because  the  group  to  be  rated  is  more  homogenous 
than  an  average  group  of  employees  (premise  that  college 
graduates  have  been  through  a longer  selective  process 
than  an  average  group  of  citizens)  a periodical  evalua- 
tion of  traits  is  maintained.  A score  of  "10"  denotes 
the  best  rating  and  "5”  the  lowest  possible  score.  The 
aim  is  to  secure  a relative  evaluation  of  each  young 
man.  Ratings  of  individual  qualities  are  given  prime 
consideration.  Management,  however,  employs  a composite 
score  in  order  to  inform  the  individual  how  he  "rates" 
in  respect  to  his  group.  All  employees  under  this  in- 
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temship  plan  are  rated  by  their  supervisors  and  are  told 
of  their  progress.  Steps  are  taken  to  see  that  the  em- 
ployee improves  on  those  traits  in  which  he  appears  be- 
low standard.  Those  who  do  not  show  sufficient  improve- 
ment by  the  next  rating  (six  months)  are  dropped  from 
the  concern.  Another  set  of  ratings  is  made  at  the  end 
of  the  first  year.  Ratings  are  continued  for  some  years 
after  completion  of  the  training  course. 

An  attempt  is  made  to  hire  those  men  who  seem  to 
fit  the  pattern  of  previous  successful  applicants.  The 
initial  interviewer  tries  to  direct  the  interview  toward 
those  traits  deemed  important  by  the  concern.  No  sta- 
tistical reports  are  given  in  the  literature  or  in  cor- 
respondence to  the  writer  on  the  subject  of  reliability 
or  validity.  Over  1,000  college  graduates,  nevertheless, 
have  completed  the  internship  of  which  over  600  are  still 
employed  by  the  concern.  A considerable  proportion  hold 
high  positions. 

Company  B . — This  company  has  done  a great  deal  of 
constructive  work  on  rating  scales.  Merit  rating  is 
deemed  necessary  in  order  to  adequately  handle  the  com- 
pany’s personnel.  This  concern  employs  a graphic  type 
of  scale  and  uses  fourteen  traits.  The  selection  of 
traits  was  on  the  basis  of  a number  of  qualities  or  traits 
considered  important  in  most  Jobs.  Construction  and 
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standardization  of  headings  under  each  trait  were  carried 
out*  No  attempt  is  made  to  obtain  a composite  rating* 

A different  sheet  is  used  for  each  trait*  This  is  one 
of  the  few  concerns  that  has  done  a great  deal  regarding 
the  scientific  accuracy  of  its  methods* 

It  is  interesting  to  note  that  this  concern  empha- 
sizes practical  application  rather  than  any  possible 
theoretical  values.  Those  men  that  handle  the  material 
from  the  practical  side  must  consider  one  rating  at  a 
time.  Reliable  group  statistics  may  be  of  little  value 
in  interpreting  the  results  for  any  one  individual. 

The  qualitative  aspects  of  rating  are  constantly  stress- 
ed. When  ratings  are  treated  statistically,  they  may 
become  meaningless  and  lose  their  practical  value  even 
though  they  gain  in  appeal  to  the  judges. 

Correlations  between  judges  range  from  *62  to  .90. 

A frequent  difficulty  of  industrial  ratings  is  that  Jud- 
ges try  to  arrive  at  the  same  conclusions  to  the  detri- 
ment of  accuracy.  Often  foremen  cooperatively  try  to 
keep  their  ratings  nearly  the  same  because  the  results 
play  such  an  important  part  in  the  pay  rate  or  bonus  of 
the  employee.  This  company’s  emphasis  on  knowing  the 
individual  as  well  as  possible  both  as  to  assets  and 

liabilities  is  most  encouraging.  Recognition  by  manage- 

• 

ment  that  no  merit  rating  program  can  work  unless  there 
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Is  sufficient  energy  and  active  interest  is  timely. 

Company  C. — Another  large  concern  with  numerous 
subsidiaries  has  employed  rating  methods  to  good  advan- 
tage. Supervisors  at  this  concern  are  rated  by  a plan 
they  themselves  devised.  They  wanted  the  benefit  of  a 
review  of  personal  traits  in  addition  to  performance 
factors.  Small  discussion  groups  gave  each  man  a chance 
to  suggest  factors  that  should  be  considered  in  scoring 
job  performance  and  personality.  Then  the  group  as  a 
whole  decided  which  points  were  to  be  included.  A graphic 
scale  of  twelve  traits  on  a five  degree  basis  is  the  in- 
strument adopted.  Space  is  provided  where  the  rater 
may  suggest  methods  of  helping  an  employee  to  be  more 
effective  in  his  Job.  Ratings  are  administered  once  a 
year.  This  concern  has  found  this  plan  beneficial  with- 
out the  use  of  any  statistical  devices. 

Company  D. — A large  financial  organization  has  de- 
vised a graphic  type  of  rating  scale  which  has  proven 
helpful.  Success  in  the  organization  is  estimated  as 
being  determined  by  the  degree  to  which  the  employee 
possesses  various  traits.  Each  trait  is  described,  and 
the  supervisor  is  required  to  rate  all  members  of  the 
group  at  the  same  time.  All  the  men  under  the  direction 
of  a particular  foreman  or  supervisor  are  listed  on  one 
sheet.  In  Company  D correlations  between  first  and  second 
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ratings  range  from  ,48  to  .72  and  between  second  and 
third  ratings  from  .54  to  .79. 

Company  E. — This  concern  employs  psychological 
tests  and  personality  ratings  in  their  personnel  work. 

The  scale  is  the  graphic  type  and  consists  of  fifteen 
traits  on  a five  degree  spread.  Points  extend  from  zero 
to  one  hundred  for  each  trait.  The  company  has  included 
both  performance  and  personality  traits  on  the  same  scale. 
The  traits  listed  are  (1)  supervision  required,  (2)  atti- 
tude toward  supervisors,  (3)  application  to  work,  (4) 
quality  of  work,  (5)  attendance,  (6)  dependability,  (7) 
intelligence,  (8)  cooperation,  (9)  initiative,  (10)  am- 
bition, (11)  personal  qualities,  (12)  versatility,  (13) 
leadership,  (14)  knowledge  essential  to  supervisor's 
job,  and  (15)  capacity  to  advance. 

This  is  one  of  the  few  companies  that  has  shown 
real  progress  in  constructing  a manual  for  the  purpose 
of  explaining  the  "raison  d'etre"  of  ratings,  definitions 
of  terms,  and  rules  for  Judges.  A composite  rating  is 
maintained  unless  the  employee  has  fallen  below  the  score 
of  55  on  any  of  the  attributes.  Descriptions  are  divid- 
ed into  five  classifications.  An  over-all  evaluation 
is  also  presented  and  falls  into  six  groups;  (1)  satis- 
factory: ability  beyond  present  Job;  should  be  advanced 
when  opportunity  presents  itself,  (2)  satisfactory:  a 
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good  employee;  can  probably  advance  with  more  experience 
and  training,  (3)  satisfactory:  but  appears  to  have  reach- 
ed limit,  (4)  unsatisfactory:  but  with  more  training 
and  experience  should  become  satisfactory,  (5)  unsatis- 
factory: should  be  considered  for  transfer  for  the  fol- 
lowing reasons,  (6)  unsatisfactory:  recommended  for  re- 
lease, Space  is  provided  for  remarks  of  the  rater  and 
comments  of  the  direct  supervisor  and  department  head. 

Use  of  Rating  Scales  in  Education 

In  the  school  situation  rating  scales  described 
in  the  literature  were  read.  Some  scales  were  obtained 
from  the  publishers.  Others  were  obtained  by  writing 
to  certain  high  schools  and  to  the  State  Departments  of 
Education  for  copies  of  their  rating  scales  and  instruc- 
tions for  their  use.  A description  of  some  representa- 
tive types  of  rating  scales  used  in  the  field  of  educa- 
tion at  the  secondary  level  follows. 

The  BEO  Personality  Rating  Schedule. — This  is  one 
of  the  instruments  used  by  the  Business  Education  Coun- 
cil, which  represents  the  National  Office  Management 
Association  and  the  Eastern  Commercial  Teachers  Associa- 
tion. The  Council  certifies  students  who  have  pursued 
studies  in  preparation  for  office  work.  The  scale  may 


. . -ymj •.•arryo  mor  'I'w  ■orsvbs  y r;  t.-irnr  r.o  ; Zq::.e  roog 
i:'.  evsri  od  B'lsonga  du  cf  ly'todc.B'V  [ . ) t cls^d  -brtfl 

: « 

- • t • r • 

: : ''lOdOSl 

- J ' • ’ : ' • 

• ■'.:•/•■•  ©rid  lo  Bi(*xBflje*x  10  ' £ f • • ' - 5 - -- 

. ' • ■ 1 r ■ 

f:d  OD r ft  8 i ' 1 1 ":r  JU  . //_  c J 

■ 

.b£&i  ©*i©w  enuds'iedlJ  ©rid  nl 

* ' 

'to  ednemdnsceC  ©d*d£  ^rid  od  f*na  eloorion  d^lri  aisd-ieo  od 
_ • ' ' io*  1 

-sd  ft©  b©*i  gen  sroe  *0  rrc  idq  f'-a  £v,d 

-sou 5©  lo  M©!**  ©rid  nl  beau  aelsoa  gnldsn  lo  aeqyd  evld 

. ••  r'  '•  lev  I ^ *rid  ds  acid 

fjfio  al  eld?  — — * ©J  ... 

-r-uoC  ri  .?••!  eouJ&S  eaenlfliifl  ©rid  yd  &*ea  edfleirxvid'v-'i  • id 

- ■ ■ • Islo*  I c 

• noli 

yarn  ©Isce  ©riT  . ■ ! Bold  8cy©*tq  rrl  eelbudo 


be  used  from  grades  seven  to  sixteen. 

There  are  eight  principal  scales  in  the  schedule, 
and  for  each  of  these  there  are  several  sub-scales. 

The  eight  principal  scales  are  mental  alertness,  initia- 
tive, dependability,  cooperativeness,  judgment,  personal 
impression,  courtesy,  and  health.  A.  five  point  graphic 
type  of  scale  is  employed.  The  score  for  each  principal 
scale  is  computed  by  averaging  the  ratings  on  appropriate 
sub-scales.  No  reliability  or  validity  coefficients 
are  mentioned  in  the  manual  accompanying  the  scale  nor 

1/ 

in  a published  article  by  one  of  the  authors. 

Scale  for  Evaluating  the  School  Behavior  of  Chil- 

2/ 

dren  Ten  to  Fifteen.—  This  scale  developed  by  Hayes 
consists  of  one  hundred  statements  about  behavior  which 
are  arranged  in  eight  groups  as  follows:  relation  to 
teacher,  respect  for  right  of  others,  relation  to  other 
pupils,  initiative,  health  habits,  general  interests, 
scholarship,  and  study  habits.  The  instrument  was  de- 
veloped primarily  for  use  by  teachers  in  rating  pupils 
but  may  also  be  used  by  pupils  for  self-evaluation. 

The  author  reports  a reliability  of  .94  for  the  scale. 

1/  P.  J.  Rulon,  "A  Personality  Rating  Schedule",  Harvard 
Teachers  Record:  1936:  6:  46-53* 

2/  M.  Hayes,  "A  Scale  for  Evaluating  Adolescent  Person- 
ality", Pedagogical  Seminary  and  Journal  of  Genetic 
Psychology:  1934:  44:  206-222. 
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Data  indicate  that  the  scale  differentiates  between 
desirable  and  undesirable  personalities. 

Phoenix  Union  High  School. — This  school  has  a sen- 
ior rating  chart  which  consists  of  ten  personal  traits 
and  eight  special  talents.  The  traits  are  as  follows: 
dependability,  cooperation,  effort,  honesty,  initiative, 
personal  appearance,  punctuality,  emotional  stability, 
health,  and  work  habits.  Traits  are  checked  high,  av- 
erage, or  low.  There  are  three  lines  of  space  available 
for  general  remarks.  Reliability  and  validity  of  the 
Instrument  Is  not  discussed. 

1/ 

Rating  of  Personal  Traits. — Miller  has  developed 
this  graphic  rating  scale  which  lists  13  personal  traits. 
The  author  states  that  the  scale  may  be  used  at  the  ele- 
mentary and  high  school  level.  If  ratings  begin  at  the 
former  level,  they  may  be  used  for  preventative  or  cor- 
rective action  when  necessary.  The  author  also  points 
out  practical  ways  of  using  the  information  gathered  on 
the  rating  sheet.  He  mentions  that  the  first  six  traits 
are  ones  generally  agreed  upon  as  being  Important  for 
all  Jobs.  The  remaining  seven  traits  are  considered 
of  value  in  making  adjustments  in  school  situations, 
society,  and  certain  specialized  occupations.  The  traits 
are  (1)  cooperativeness,  (2)  trustworthiness,  (3)  Judg- 

1 / L.  M.  Miller,  Rating  of  Personal  Traits.  The  Chron- 
icle. Fort  Byron,  New  York,  19^1,  2pp. 
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ment,  (4)  initiative,  (5)  industriousness,  (6)  appear- 
ance, (7)  health,  (8)  voice  and  enunciation,  (9)  emo- 
tional reaction  type,  (10)  personality  status,  adjustment, 
integration,  (11)  leadership,  (12)  intellectual  curiosi- 
ty, (13)  reading.  No  information  as  to  the  reliability 
and  validity  of  the  scale  is  listed. 

Dobbs  Ferry  High  School. — The  personality  trait 
chart  of  this  school  consists  of  11  traits  which  are 
accuracy,  cooperation,  effort,  attitude,  reliability, 
initiative,  leadership,  self-control,  voice  enunciation, 
personal  appearance,  and  courtesy.  Each  trait  is  divid- 
ed into  a scale  which  extends  from  one  to  five.  The 
key  is  as  follows:  (1)  excellent,  (2)  good,  (3)  average, 
(4)  poor,  (5)  very  poor.  Summaries  of  the  ratings  for 
each  pupil  are  placed  on  a permanent  record  card.  The 
scale  is  to  be  used  from  the  seventh  through  the  twelfth 
grades.  The  author  does  not  mention  reliability  and 

validity  coefficients  for  the  scale  either  in  a publish- 

1/ 

ed  article  or  in  a letter  to  the  writer.  Heuss  does 
mention,  however,  that  the  use  of  rating  charts  has  prov- 
en beneficial  to  pupils.  Her  emphasis  that  rating  scales 
have  as  definite  a place  in  the  school  as  they  do  in  in- 
dustry has  furthered  their  use. 

l/  C.  Heuss,  ,yWe  Introduced  Rating  Charts",  Occupations : 
1947:  25:  216-219. 
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The  Senior  High  School  of  Kalamazoo.  Michigan. -- 


This  school  has  a rating  scale  printed  on  the  cumulative 
record.  The  traits  are  cooperation,  dependability,  ex- 
ecutive ability,  emotional  control,  Industry,  initiative, 
integrity,  persistence,  and  personal  appearance.  Ra- 
tings are  checked  one,  two,  or  three  corresponding  to 
high,  average,  or  low.  There  is  very  little  space  for 
anecdotal  records.  No  information  is  given  as  to  the 
reliability  and  validity  of  the  scale. 

Minneapolis  Public  Schools. — At  the  high  school 
level  rating  sheets  are  employed  for  cooperative  part- 
time  work  students  in  the  distributive  trades.  Three 
forms  are  used.  The  employer  rates  the  students  for 
the  first  two  six-week  periods  on  a basis  of  satisfac- 
tory and  unsatisfactory.  At  the  end  of  the  third  six- 
week  period,  which  is  the  end  of  the  semester,  a final 
letter  grade  is  given  to  summarize  the  employer's  final 
rating . 

One  of  the  forms  lists  the  following  traits,  (l) 
health,  (2)  appearance,  (3)  manner,  (4)  initiative,  (5) 
industry,  (6)  accuracy,  (7)  loyalty,  (8)  cooperation, 

(9)  responsibility,  and  (10)  knowledge.  On  each  line 
a check  should  be  placed  over  the  phrase  which  describes 
the  worker.  The  chart  extends  on  a five  point  scale. 
Traits  one,  four,  six,  and  eight  decrease  from  excellent 
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to  deficient  while  the  other  traits  increase  from  defi- 
cient to  excellent.  Space  is  available  for  additional 
comments.  No  mention  of  reliability  or  validity  has 
been  found  by  the  writer  in  the  literature  nor  in  cor- 
respondence with  the  Minneapolis  Public  Schools. 

The  Haggerty-Olson-Wlckman  Behavior  Rating  Sched- 

V 

ules.—  The  scales  were  designed  for  the  study  of  be- 
havior problems  and  problem  tendencies  in  children. 

In  Schedule  A the  rater  checks  each  item  to  indicate 
the  relative  frequency  of  occurrence  of  the  behavior 
in  the  person  being  rated.  Schedule  B is  a graphic  rat- 
ing scale  for  intellectual,  physical,  social,  and  emo- 
tional traits.  Twenty-four  traits  are  listed,  and  each 
one  is  described  on  a five  point  scale. 

An  intensive  study  of  the  reliability  and  validity 
of  the  instrument  was  confined  to  Schedule  B.  The  re- 
liability of  Schedule  B has  been  reported  in  various 
studies  with  correlation  coefficients  varying  from  .56 
to  .92  in  nursery,  elementary,  and  high  school  popula- 
tions • 

The  validity  of  the  Behavior  Rating  Schedules  has 
been  studied  by  means  of  clinical  case  ratings  and  sub- 
sequent histories  of  children.  Validity  coefficients 

1/  W.  G.  Olson,  "The  Clinical  Use  of  Behavior  Rating 
Schedules",  Journal  of  Juvenile  Research:  1931 s 15s 
237-245. 


' < ■■  ■ • 

• ' 

: 

D'xiiis'ied'lX  er  ct  nl 

. " r ■ V C"  ' ' ' r ■ 

' ' “ - 

' 

' — • 

iclvBtiod  srld  tc  •oneTUaooo  lo  yojfWJ po*i^t  evtSBlo*. t eritf 

• . ■ 

' • ■ . - ' 

. ■ 

ywXMXav  bne  y^IIXcfsIXe'x  erJJ  *o  y^u^e  ~nl  aA 

- ■ 

bjjoX'IJBV  nX  rrsecf  esri  9 ©Xubfcriofi  lo  y^XXXcfBlI 

' . . * 

. Pi 

. 

ib  E^nU’sri  eeao  l£-  : o rrtseai  ycf  Jb' 

. 

« • 

: ; ' . 

. 


vary  from  .60  to  .7 6 


Reliability  and  Validity 

Methods  of  measuring  reliability. — Although  relia- 
bility and  validity  will  be  discussed  in  detail  as  it 
pertains  to  the  Vocational  Service  Rating  Scale  in  Chap- 
ter V,  some  information  is  now  presented  as  background. 
There  is  considerable  variation  in  the  results  reported. 
Reliability  and  validity  indices  are  not  mentioned  in 
some  reports.  With  the  exception  of  some  very  large  con- 
cerns, statistical  measures  for  determining  reliability 
and  validity  are  for  the  most  part  forgotten  or  inade- 
quate . 


,1/ 

Garrett's  definition  of  reliability  is  a workable 
concept.  "The  reliability  of  a test  or  of  any  measuring 
instrument  depends  upon  the  consistency  with  which  it 
gauges  the  ability  of  those  to  whom  it  is  applied." 

In  measuring  reliability  of  rating  scales  the  pre- 
valent methods  have  been  (1)  to  have  the  same  Judges  rate 
individuals  again  at  a later  time;  (2)  to  correlate  one- 
half  of  an  instrument  with  the  other  half;  and  (3)  to 
compare  ratings  by  different  Judges  on  the  same  individual 

These  three  methods  of  determining  reliability  fur- 
nish estimates  rather  than  strictly  accurate  measures 

1/  H.  E.  Garrett , Statistics  in  Psychology  and  Education. 
Longmans  Green  and  Company.  New  York,  1938,  second  edi- 
tion,  p.  311. 
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of  the  consistency  of  the  test  scores.  Each  of  the  meth- 
ods is  subject  to  certain  qualifications,  and  each  holds 
good  under  certain  conditions* 

If  a rating  is  repeated  within  a short  period  of 
time,  many  raters  will  undoubtedly  recall  their  first 
scores  and  be  influenced  by  them* 

Because  of  the  difficulty  in  knowing  and  con- 
trolling the  conditions  which  influence  the  scores 
on  the  second  administration  of  a test,  the  repeti- 
tion method  of  determining  test  (or  rating  scale) 
reliability  is  less  generally  used  than  the  other 
two  methods*  l/ 

The  repetition  method  is  not  acceptable  if  one  concurs 

with  the  premise  that  individuals  can  change  their  point 

scores  through  time  and  effort. 

In  correlating  parts  of  an  instrument,  relatively 

temporary  attitudes,  feelings,  and  the  like,  which  may 

differ  at  another  time  and  thus  cancel  out,  affect  the 

scores  on  both  halves  of  the  test  in  the  same  way.  This 

2/ 

tends  to  make  the  coefficient  of  reliability  too  high. 

The  third  method,  which  avoids  the  disadvantages  of 
the  other  two  methods  and  the  one  used  in  this  research, 
is  to  compare  ratings  by  different  Judges  on  the  same 
individual.  We  know  that  the  average  of  five  determina- 
tions of  height,  for  example,  is  more  reliable  than  a 
1/  Ibid,  p.  312. 

2/  Ibid,  p.  313* 
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single  determination. 


This  fact  has  also  been  shown 


in  the  field  of  ratings  with  qualified  Judges. 

2/ 

Rugg  reports  that  for  several  hundred  ratings  of 
ten  teachers  by  a dozen  trained  school  officers,  no  co- 
efficient of  reliability  exceeded  0.2,  and  many  were  neg- 

2/ 

atlve.  Swab  and  Peters  found  a correlation  of  .98  when 
30  pupils  in  the  seventh  grade  and  34  in  the  eighth  grade 
of  a small  Pennsylvania  school  system  ranked  all  other 
pupils  in  each  grade  on  the  traits  of  the  test.  In  Nan- 

hJ 

nlga's  study,  where  reliabilities  of  .60  and  .70  are 
reported,  30  pupils  in  a Junior  high  school  were  rated 
by  five  teachers  on  15  traits  on  a five  degree  scale. 

The  test  was  repeated  in  two  weeks  with  the  order  of  the 

5/ 


traits  changed.  Uhrbrock  had  353  Judges  rate  137  col- 
lege seniors.  He  estimated  the  reliability  of  the  Judges 
1/  Ibid,  p.  315. 

2/  H.  0.  Rugg,  "Is  the  Rating  of  Human  Character  Prac- 
ticable?", Journal  of  Educational  Psychology:  1921  i 12: 


425-438 . 

2/  J.  C.  Swab  and  C.  C.  Peters.  "The  Reliability  and 
Validity  of  Estimates  (Ratings)",  Journal  of  Educational 
Sociology:  1933:  7:  224-232. 

4/  S.  P.  Nanniga,  "A  Critical  Study  of  Rating  Traits", 
Educational  Administration  and  Supervision:  1926:  12: 
114-119. 

5/  R.  S.  Uhrbrock,  "Rating  Tendencies  of  Personally  Se- 
lected Judges",  Journal  of  Educational  Psychology:  1932: 
23:  594-603. 
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1/ 

rating  to  be  .73*  Chi  reports  that  the  average  relia- 
bility coefficient  of  each  teacher  rating  and  rerating 

100  pupils  in  the  fifth  and  sixth  grades  of  elementary 

2/ 

school  is  .81*  Stevens  and  Wonderlic  report  correlations 

from  *48  to  *79  on  a particular  rating  scale.  This  scale 

was  used  for  over  two  years  in  a business  organization 

and  included  ratings  on  more  than  1,900  subjects.  Van 

2/ 

Alstyne,  reporting  a scale  for  rating  school  behavior, 
reports  correlations  from  .72  to  .82.  She  had  201  children 
re rated  at  intervals  of  two  to  eight  weeks  toward  the 

v 


end  of  the  school  year.  Driver  in  determining  the  re- 
liability of  a 14  trait  rating  scale  compared  the  ratings 
for  two  successive  years  of  one  Judge  on  188  to  300  em- 
ployees. Coefficients  of  correlation  ranged  from  a low 
of  .59  to  a high  of  .86. 

1/ 


Methods  of  measuring  validity.—  Garrett  clearly 

S7TT  L.  Chi,  "Statistical  Analysis  of  Personality  Ra- 
ting", Journal  of  Experimental  Education:  1937?  5?  229- 
245. 

2/  S.  N.  Stevens  and  E.  F.  Wonderlic,  "An  Effective  Re- 
vision of  the  Rating  Technique",  Personnel  Journal:  1934; 
13:  125-134. 


D.  Van  Alstyne,  "A  New  Scale  for  Rating  School  Behav- 
ior and  Attitudes  in  the  Elementary  School",  Journal  of 
Educational  Psychology:  1936:  27:  677-693- 

4/  R.  S.  Driver,  "A  Case  History  in  Merit  Rating",  Per- 
sonnel: 1940:  16:  151-132. 

Ji/  H.  E.  Garrett,  op.  cit.,  p.  324. 
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defines  the  term  validity:  "The  validity  of  a test,  or 
other  measuring  instrument , depends  upon  the  fidelity 
with  which  it  measures  whatever  it  purports  to  measure." 

In  measuring  the  validity  of  rating  scales  three 
methods  often  used  are  (1)  to  compare  ratings  with  outside 
criteria;  (2)  to  employ  the  index  of  reliability;  and  (3) 
to  compare  ratings  by  different  Judges  on  the  same  indi- 
vidual. 

The  use  of  an  objective  measure  of  performance  as 
a criterion  is  the  only  direct  method  of  determining  the 
validity  of  a rating  scale.  Production  on  the  Job  is 
the  most  obvious  criterion.  Others  may  be  estimates  as 
to  ability  in  the  Job  by  an  employee's  supervisor,  work 
samples,  breakage,  accident  claims,  and  amount  of  mater- 
ial wasted. 

i/ 

Burtt  cites  a study  where  the  validity  is  good. 

The  criterion  was  available  in  the  case  of  salesmen  whose 
annual  commission  earnings  were  provided.  In  two  analyses 
he  reports  that  multiple  correlation  coefficients  of  .66 

and  .85  were  obtained  between  ratings  and  other  records. 

2/ 

McMurry  and  Johnson  report  a validity  coefficient 

l/  H . E . Burtt , Principles  of  Employment  Psychology.  Har- 
per & Bros.  New  York,  1942,  revised  edition,  p.  385* 

2/  R.  N.  McMurry  and  D.  L.  Johnson,  "Development  of  In- 
struments for  Selecting  and  Placing  Factory  Employees", 
Advanced  Management : 1945s  10:  120. 
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of  .68  with  prediction  of  job  success. 

Comparative  rating  techniques,  when  properly 
handled  provide  useful  and  consistent  criterion 
evaluations.  Supervisors*  ratings  showed  fairly 
high  Intercorrelations  with  each  other  and  with 
ratings  based  on  production  records. 


Numerous  investigators,  however,  doubt  whether  valid- 
ity coefficients  for  ratine  scales  can  be  obtained.  This 

1/ 

opinion  is  expressed  by  Driver. 


The  problem  of  ascertaining  that  the  ratings 
actually  measure  what  they  are  presumed  to  measure 
is  the  most  difficult  factor  in  the  whole  procedure. 
The  difficulty  arises  in  the  main  from  the  fact  that 
if  we  had  some  criterion  with  which  to  compare  the 
results  of  our  ratings,  in  all  probability  we  would 
make  direct  use  of  this  measurement  and  disregard  the 
attempt  to  obtain  ratings.  Actually,  however,  for 
many  of  the  traits  rated  there  are  no  objective  cri- 
teria readily  available.  Thus  the  problem  of  deter- 
mining that  the  ratings  measure  what  they  are  supposed 
to  measure  is  well  nigh  unsolvable. 

2/ 

Smyth  and  Murphy  suggest  that  those  who  set  up  ra- 
tings are  generally  satisfied  with  "logical  validity"  as 
against  "statistical  validity." 

"it  is  evident  that  to  demonstrate  that  a set 
of  ratings  is  valid  there  must  be  some  objective 
criterion  of  effectiveness  of  performance  against 
which  the  ratings  may  be  compared.  However,  since 
the  ratings  are  normally  used  because  there  are  no 
such  criteria,  it  is  obvious  that  those  who  set  out 
to  try  to  prove  that  ratings  are  valid  soon  find 
themselves  going  about  in  the  proverbial  vicious 
circle . 

l/  R.  S.  Driver,  "A  Case  History  in  Merit  Rating",  Per- 
sonnel: 1940:  16:  153-154. 


2/  R.  C.  Smyth  and  M.  J.  Murphy,  Job  Evaluation  and  Em- 
ployee Rating.  McGraw  Hill.  New  York,  1946,  p.  189* 
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"Although  the  question  of  measuring  validity 
has  been  of  concern  to  many  of  those  interested 
in  rating  methods,  those  individuals  have  not  re- 
frained from  developing  and  using  rating  plans  be- 
cause of  the  absence  of  simple  and  clear-cut  solu- 
tions to  the  problem.  Recognizing  the  inevitabil- 
ity of  some  type  of  rating  and  desiring  to  control 
and  systematize  the  process  those  who  set  up  rating 
plans  commonly  are  satisfied  with  what  for  want 
of  a better  name  may  be  called  'logical  validity* 
in  contrast  with  'statistical  validity'.  Logical 
validity  means,  in  essence,  the  plain  reasonable- 
ness and  common-sense  significance  of  the  traits 
that  are  included  in  the  rating  form  and  of  the 
weightings  that  are  assigned  to  them...."  l/ 

The  index  of  reliability,  a second  method,  gives  the 

correlation  between  obtained  scores  and  theoretically  true 

scores  in  the  same  function.  The  index  is  in  reality  a 

validity  coefficient  since  it  tells  how  well  a test  is 

measuring  true  ability  in  the  function  which  it  purports 

to  measure.  The  index  of  reliability  is  one  of  the  means 

used  to  measure  the  validity  of  the  Vocational  Service 

Rating  Scale. 

i/ 

In  their  Studies  on  Deceit,  Hartshome  and  May  re- 
port a correlation  of  between  *50  and  .60.  These  investi- 
gators used  the  index  of  reliability  in  determining  the 
theoretical  validity  of  certain  deception  tests. 

A third  method  of  determining  validity  and  a pro- 
1/  Ibid. 

2/  R.  Hartshome  and  M.  May,  Studies  in  Deceit.  Macmil- 
lan Co.  New  York,  1928,  Part  II,  p.  126>. 
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cedure  followed  in  this  study  is  to  compare  the  ratings 
by  different  judges  on  the  same  individual.  This  is  also 
a method  of  determining  reliability  and  was  described 
on  page  43. 

1/ 

Rogers  compared  the  ratings  by  various  Judges  to 
determine  the  validity  of  the  scales  he  used.  He  reports 
validity  coefficients  of  .59  for  inferiority,  .52  for 
social  maladjustment,  and  .45  for  general  emotional  ad- 
justment. 

2/ 

Remmers  summarizes  the  thoughts  of  the  writers  who 
consider  reliability  and  validity  synonymous  in  the  case 
of  rating  scales. 

...While  reliability  may  be  defined  as  the 
accuracy  with  which  a measuring  instrument  measures 
whatever  it  does  measure,  validity  is  defined  as 
the  extent  to  which  the  instrument  measures  what 
it  purports  to  measure.  Since  it  is  student  Judg- 
ments that  constitute  the  criterion,  reliability 
and  validity  are  in  this  case  synonymous.... 

In  reading  the  literature  one  finds  that  many  in- 
vestigators have  used  more  than  one  method  in  establishing 
the  validity  of  rating  scales.  This  seems  necessary  and 
appropriate  in  view  of  the  fact  that  the  whole  problem 

V -■  R.  Rogers , Measuring-  Personality  Adjustment  of  Chil- 
dren Nine  to  Thirteen  Years  of  Ape . Teachers  College, 
Columbia  University,  New  York,  1941,  Contributions  to 
Education,  vol.  458,  107  pp. 

2/  H.  H.  Remmers,  "Reliability  and  Halo  Effect  of  High 
School  and  College  Students'  Judgments  of  Their  Teachers", 
Journal  of  Applied  Psychology:  1934:  18:  621. 


of  validating  personality  scales  is  still  very  much  in 
a state  of  flux* 


Sources  of  Error  in  Rating  Procedure 
Halo.—  Halo  assumes  that  if  the  rater's  general 
Impression  of  some  particular  trait  or  aspect  of  the  per 
sonality  is  received  favorably,  there  is  a tendency  to 
overestimate  the  desirable  and  underestimate  the  undeslr 
able  in  his  personality.  The  converse  is  also  assumed 
to  be  true. 

1/ 

Johnson  strikes  a strong  blow  against  the  halo  con 
cept  as  it  is  generally  conceived. 

This  concept  has  worked  its  way  into  techni- 
cal psychological  literature  and  even  into  elemen- 
tary text  books,  but  anyone  who  looks  for  clear- 
cut  evidence  on  the  halo  effect  is  certain  to  be 
disappointed.  The  evidence  usually  given  comes 
from  correlations  between  ratings  of  personality 
traits  since  confusion  between  the  separate  traits 
and  the  general  impression  will  increase  the  cor- 
relation between  the  separate  traits ... .The  diffi- 
culty with  such  evidence  is  that  correlations  be- 
tween ratings  are  ambiguous.  It  is  not  easy  to 
decide  whether  the  correlation  is  the  result  of 
a confusion  in  the  mind  of  the  rater  or  of  an  ob- 
jective relationship  within  the  variables  being 
rated .... 

Symonds  approaches  the  problem  with  the  technique 

of  partial  correlation  in  an  attempt  to  eliminate  the 

effect  of  the  general  impression  and  thus  to  ascertain 

its  importance.  He  had  two  teachers  rate  their  pupils 

TTo7  M.  Johnson,  "A  Systematic  Treatment  of  Judgment", 
Psychological  Bulletin:  1945 1 42:  203* 
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on  seven  personality  traits,  A composite  rating  was 
obtained  by  adding  the  seven  separate  ratings  for  each 
teacher's  general  impression.  Partial  correlations  of 
the  second  order  were  computed  to  find  the  relation  be- 
tween the  two  ratings  of  each  separate  trait  with  the 
influence  of  each  teacher's  general  impression  ruled 
out.  The  main  difference  of  .24  was  taken  as  an  indi- 
cation of  the  halo  effect. 

1/ 

Johnson  refutes  Symonds'  concept  of  halo. 

"Yet  this  argument  is  open  to  the  same  criti- 
cism as  the  others,  namely  that  the  correlation 
may  be  in  the  people  rather  than  in  the  rating  pro- 
cess. If  Symonds  had  begun  with  objective  test 
scores,  a similar  set  of  correlations  might  have 
been  obtained.  The  correlations  between  the  sep- 
arate test  scores  would  be  considered  reliability 
coefficients  and  the  correlations  between  each 
test  and  battery  would  be  considered  a measure  of 
the  influence  of  the  general  factor  on  each  test. 
The  reliability  of  each  test  would,  of  course,  be 
lowered  when  the  general  factor  is  partialled  out. 
Hence,  it  is  not  necessary  to  assume  any  halo  ef- 
fect to  account  for  these  results.  Thus  it  appears 
that  the  evidence  for  the  halo  effect  is  rather 
thin. 


"Although  everyone  who  has  had  any  experience 
with  ratings  of  personality  acknowledges  the  impor- 
tance of  this  phenomenon,  it  would  be  well  to  have 
objective  data  on  its  incidence  and  amount. 

"The  best  technique,  the  writer  would  suggest 
for  the  present,  is  manipulation  of  the  procedure 
of  Judging  rather  than  of  the  data  obtained  by  one 
procedure . . . ." 

The  writer  in  carefully  reading  the  literature  is 
unable  to  find  any  conclusive  experimental  evidence  as 
1/  Ibid. 
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to  the  size  of  the  coefficients  which  would  establish 
the  existence  or  extent  of  "halo"  effect.  On  an  em- 
pirical level  one  should  expect  that  there  should  be  a 
certain  correlation  between  desirable  traits  of  any 
type  and  conversely  for  negative  traits.  Traits  do  not 
operate  alone.  They  are  regarded  as  operating  alone, 
however,  in  our  efforts  to  study  personality. 

Tendency  toward  leniency. — Another  possible  limi- 
tation of  ratings  is  what  is  described  as  the  tendency 
toward  leniency  on  the  part  of  the  judges.  Many  raters 
especially  in  the  industrial  situation  hesitate  to  give 
adverse  ratings  because  they  realize  how  important  a 
part  they  play  in  the  job  security  of  the  individual 
employee . 

Error  of  central  tendency. — Another  similar  limi- 
tation is  the  error  known  as  the  error  of  central  tenden- 
cy. It  is  the  disposition  to  rate  as  average  all  those 
who  are  not  well  known  to  the  rater.  This  limitation 
is  overcome  by  making  certain  that  raters  are  continual- 
ly cognizant  of  the  traits  of  the  men  under  their  super- 
vision. When  sufficient  observation  is  lacking,  many 
scales  now  allow  raters  to  mark  a trait,  "no  opportunity 
to  observe." 

Summary 

A history  of  rating  scales  shows  that  they  may  be 
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an  extremely  valuable  tool  if  there  is  a constant  criti- 
cal appraisal  of  judgments  and  adequate  training  of  ra- 
ters. In  the  Industrial  situation  there  has  been  a steady 
increase  in  the  use  of  rating  scales.  In  the  schools 
there  has  been  a tendency  for  report  cards  to  include 
an  evaluation  of  trait  ratings  and  conduct  habits  as  well 
as  traditional  grades. 

1/ 

Moore  summarizes  the  acceptance  of  rating  scales 
in  personnel  work. 

There  is  no  doubt  today  about  the  value  and 
effectiveness  of  well-constructed  rating  scales. 

They  have  become  an  integral  part  of  personnel  work, 
and  are  used  more  widely  than  any  other  one  tool 
that  has  been  the  product  of  psychological  research 
• • • • 

A need  for  active  cooperation  between  school  and 
industry  for  research  in  and  use  of  specific  rating  scales 
is  indicated.  Steps  are  gradually  being  taken  in  this 
direction.  A program  of  this  type  will  probably  lead 
to  greater  efficiency  in  the  construction  and  use  of 
rating  scales. 


TThT  Moore,  Psychology  for  Business  and  Industry.  McGraw 
Hill.  New  York,  1939,  p.  226. 
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CHAPTER  III 


BACKGROUND  AND  USE  OF  RATING  SCALE  MANUALS 

Use  of  Rating  Scale  Manuals  in  Industry 

Lack  of  manuals Comprehensive  industrial  rating 
scale  manuals  for  the  instruction  of  raters  are  meagre 
according  to  the  data  revealed  in  Table  2 in  the  preced- 
ing Chapter.  Of  the  35  companies  using  rating  scales 
only  five  employ  manuals.  This  is  a hazardous  procedure 

, 1/ 

especially  in  view  of  DriverX  statement  that  approxi- 
mately 90  per  cent  of  the  difficulties  with  rating  scales 
are  due  to  the  judges  while  only  10  per  cent  are  due  to 
the  rating  forms. 

Emphasis,  moreover,  is  placed  upon  the  results  of 
rating  rather  than  upon  adequate  training  of  judges. 

In  many  Industrial  concerns  elaborate  records  are  kept 
as  a basis  for  hiring,  firing,  transfer,  wage  increases, 
and  promotion,  but  detailed  instructions  for  Judges  are 
omitted. 

Although  psychologists  and  educators  have  stressed 

the  importance  of  exact  procedures  in  the  administration 

of  both  individual  and  group  tests,  this  has  been  gross- 

1/  R.  S.  Driver,  "A  Case  History  in  Merit  Rating",  Per- 
sonnel: 1940:  16:  150. 


ill  r ; *’ 


r\ 

w 


r :a  c .re  :o;  :l 


nJeubcrl  ;ri  el-simsM  el 


n..‘  f • " ?■« 


- • ' ' • ' -r  ’ . • • -i  • 

. 


’ ' V'.  I -LJ  • " ;' 


' 


. - 

. 

j.-  c-.-r  . • a’^rtcc  -i  3<j  ioo  • '3  *■  ;'oo  I < .*••••  I nl 

' 


t>688  rf^jjoritflA 

rtoJCu Rnfelnlv-ba  eriJ  rrl  ssiubeocnq  ' .•■J'xoqifll  edJ 

. ’ . : . I xttocT  to 


0'i^con  ei & erte&sfi  BrlS  rtot  elBuruui  dl&os 


.*  ' • 


- • 

- : ; - J±„ 


ly  neglected  in  the  field  of  rating  scales.  People  have 
been  rating  one  another  for  so  long  that  it  is  naively 
assumed  that  everyone  is  competent  to  do  so.  Research 
in  the  field  of  ratings  has  convincingly  shown  the  fal- 
lacy of  such  thinking. 

As  pointed  out  in  Chapter  II,  253  concerns  were  con 
tacted  for  copies  of  their  rating  scales  and  manuals 
of  instruction.  The  previous  Chapter  described  various 
types  of  scales  employed;  this  one  indicates  the  direc- 
tions available  for  raters.  The  information  gathered 
from  the  144  firms  using  rating  scales  indicated  the 
types  of  directions  or  lack  of  directions  accompanying 
the  scales.  The  examples  presented  ar6  representative 
of  the  types  seen.  The  range  of  material  varies  from 
no  instructions  to  relatively  adequate  explanations. 

Companies  without  instructions  for  raters.--  The 
only  information  for  rating  each  trait  on  the  scale  of 
a New  York  bank  is  "poor,  zero  points;  fair,  five  points; 
average,  ten  points;  good,  fifteen  points;  and  excellent 
twenty  points." 

A well  known  insurance  company  has  merely  the  fol- 
lowing statement  on  its  rating  scale:  "Rate  the  follow- 
ing qualities  using  this  scale,  A,  Excellent;  B,  Good; 

C,  Average;  D,  Fair;  E,  Poor." 

The  single  statement  on  the  rating  scale  of  a nov- 
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elty  concern  is  as  follows:  "Will  you  please  complete 
and  return  the  form  below...." 

Inadequate  instructions  for  raters. — The  rating 
directions  of  an  oil  refining  company,  electrical  equip- 
ment manufacturer,  and  a metal  fabricating  company  are 
presented  below.  They  fail  to  meet  the  five  points  neces- 
sary for  a rating  scale  manual,  which  are  presented  in 
detail  later  in  this  Chapter.  The  five  points  are  now 
summarized  as  follows:  (1)  brief  description  of  the 
background  and  need  for  rating  scales;  (2)  purpose  of 
the  particular  rating  scale;  (3)  information  as  to  the 
manner  in  which  the  scale  was  devised;  (4)  discussion  of 
reliability  and  validity;  and  (5)  instructions  for  the 
use  of  the  scale  with  adequate  definition  of  terms. 

A great  deal  of  time  and  money  is  expended  by  an 
oil  refining  company  on  the  continued  use  of  its  scale, 
the  instructions  for  which  are  as  follows: 

The  form  is  designed  to  help  you  appraise  the 
employee’s  value  to  the  organization.  Rate  the 
employee  on  the  basis  of  his  present  job.  Consi- 
der one  quality  at  a time  and  do  not  let  one  qual- 
ity influence  the  rating  of  other  qualities.  Un- 
der each  quality  there  appear  descriptive  phrases 
to  guide  you  in  determining  the  degree  of  the  trait 
present  in  the  employee.  Rate  the  employee  by 
checking  the  one  phrase  that  represents  your  Judg- 
ment of  the  employee  in  a particular  quality.  You 
must  recognize  the  importance  of  being  fair  to  the 
employee;  the  company  is  fair  to  yourself  as  a ra- 
ter. A carelessly  prepared  rating  is  of  no  value 
and  reflects  unfavorably  on  the  rater. 

A large  manufacturer  of  electrical  equipment  has 
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the  following  directions  for  its  rating  scale* 

”1*  Base  your  ratings  on  your  own  direct  know- 
ledge of  the  employee's  performance  in  the  line 
of  his  job.  Fill  in  every  item  but  indicate  on  the 
reverse  side  of  this  sheet  those  items  in  which 
you  have  only  slight  knowledge  about  this  employee* 

"2.  Concentrate  your  attention  on  one  trait 
at  a time  disregarding  the  rest  of  the  traits  for 
the  time  being.  Do  not  allow  your  judgment  on  any 
trait  to  be  influenced  by  your  general  opinion, 
whether  favorable  or  unfavorable,  of  the  person 
you  are  rating. 

"3»  In  considering  each  trait,  read  over  the 
descriptive  phrases  several  times,  thereby  fixing 
in  your  mind  the  range  of  the  scale  for  that  par- 
ticular trait.  You  will  notice  that  the  line  under 
each  trait  is  divided  into  segments.  If  the  em- 
ployee ranks  higher  or  lower  than  indicated  by  the 
descriptive  phrases,  make  use  of  the  intermediate 
segments •" 

The  rating  scale,  which  is  used  by  a metal  fabri- 
cating company,  is  very  widely  employed  in  the  metal 
trades*  The  directions  are  as  follows. 

" INSTRUCT IONS -READ  CAREFULLY 

"Each  employee's  ability  and  fitness  in  his 
PRESENT  occupation  or  for  promotion  may  be  apprais- 
ed with  a reasonable  degree  of  accuracy  and  uniform- 
ity through  this  rating  report.  The  rating  requires 
the  appraisal  of  an  employee  in  terms  of  his  ACTUAL 
PERFORMANCE.  It  is  essential,  therefore,  that  snap 
judgment  be  replaced  by  careful  analysis.  Please 
follow  these  instructions  carefully. 

"1*  Use  your  own  judgment. 

"2.  Disregard  your  general  impression  of  the 
employee  and  concentrate  on  one  factor  at  a time. 

Study  carefully  the  definitions  given 
for  each  factor  and  the  specifications  for  each 
degree • 
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M4,  When  rating  an  employee,  call  to  mind  In- 
stances that  are  typical  of  his  work  and  way  of 
acting.  Do  not  be  Influenced  by  UNUSUAL  CASES  which 
are  not  typical. 

"5.  Make  your  rating  with  the  utmost  care  and 
thought.  Be  sure  that  It  represents  a fair  square 
opinion.  DO  NOT  ALLOW  PERSONAL  FEELINGS  TO  GOVERN 
YOUR  RATING. 

"6.  After  you  have  rated  the  employee  on  all 
six  factors,  write  under  the  heading,  General  Com- 
ments, on  the  back,  any  additional  information  about 
the  employee  which  you  feel  has  not  been  covered 
by  the  rating  report,  but  which  is  essential  to  a 
fair  appraisal. 

"7*  Read  all  four  specifications  for  factor  no. 
1.  After  you  have  determined  which  specification 
most  nearly  fits  the  employee,  place  an  X in  the 
small  square  over  it.  If  the  specification  adequate- 
ly fits  the  employee,  place  an  X in  the  left  square. 
If  he  does  not  quite  measure  up  to  the  specifica- 
tion, but  is  definitely  better  than  the  specifica- 
tion for  the  next  lower  degree,  place  an  X in  the 
right  square.  Repeat  for  each  factor." 

Better  rating  manuals.—  Three  of  the  better  rating 
manuals  used  by  industry  are  those  of  a large  oil  refin- 
ing company,  a sugar  refining  company,  and  a public  util- 
ity operating  company.  They  represent  the  exception 
rather  than  the  rule.  In  all  three  concerns  a rating 
manual  accompanies  the  rating  scale.  The  manuals  of 
these  three  companies  are  discussed  briefly  below. 

The  rating  manual  of  the  large  oil  refining  company 
shows  a great  deal  of  effort  in  its  construction.  A 
brief  discussion  of  the  background  of  rating  scales  and 
why  they  are  needed  is  presented.  Further  information, 
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however,  should  be  given.  The  purpose  of  this  particular 
rating  scale  Is  presented  in  some  detail.  Objectives 
of  the  scale  are  ably  described.  Very  little  informa- 
tion is  given  as  to  how  these  objectives  will  be  obtain- 
ed. There  is  a good  deal  of  information  on  the  manner 
of  using  the  scale  and  on  general  definitions  of  terms. 
There  is  more  of  an  emphasis  on  the  mechanical  use  of 
the  scale  than  on  instructions  for  raters.  The  method 
of  devising  the  scale  as  well  as  a discussion  of  relia- 
bility and  validity  are  neglected. 

The  rating  scale  manual  of  the  sugar  refining  com- 
pany includes  the  background  of  rating  scales  but  not 
in  sufficient  detail.  A discussion  of  the  purpose  of 
this  scale  is  well  handled.  Sufficient  attention  is  not 
given  to  the  method  of  devising  the  scale.  There  is 
little  data  on  the  reliability  and  validity  of  the  scale. 
Definition  of  terms  and  instructions  for  the  judges  are 
fairly  well  handled. 

The  description  of  the  background  and  need  for  rat- 
ing scales  in  the  manual  of  the  public  utility  operating 
company  is  too  brief.  The  company,  however,  presents  a 
good  description  of  the  purpose  of  its  scale.  Informa- 
tion regarding  the  manner  in  which  the  scale  was  devised 
as  well  as  a discussion  of  reliability  and  validity  are 
extremely  limited.  Definition  of  traits  is  fairly  well 
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handled,  but  instructions  for  Judges  do  not  receive 
enough  comment. 

Use  of  Rating  Scale  Manuals  in  Education 

Lack  of  manuals. — Rating  scales  devised  for  pupils 
above  the  elementary  school  level  are  limited.  As  in 
the  industrial  situation  rating  scales  administered  by 
schools  suffer  for  the  most  part  from  a lack  of  adequate 
instructions  for  raters. 

A comparison  of  scales  obtained  from  publishers  and 
State  Departments  of  Education  and  those  described  in 
the  literature  on  rating  scales  reveal  diverse  types  and 
directions  for  Judges.  The  directions  below  are  repre- 
sentative of  different  kinds  available  in  the  school 
situation. 

Rating  scales  without  instructions  for  raters . — 

The  Dobbs  Ferry  public  school  system  has  developed  an 
intensive  well-coordinated  guidance  program  with  the  ra- 
ting scale  as  an  integral  part  of  the  picture.  Any  writ- 
ten instructions  for  its  use  are  neglected. 

Traits  on  the  senior  personal  rating  scale  of  the 
Phoenix  Union  High  School  in  Arizona  are  checked  as  to 
high,  average,  or  low.  The  single  statement  on  the  scale 
is  as  follows:  "Please  place  a check  mark  in  the  square 
which  represents  your  evaluation  of  the  student." 
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Inadequate  Instructions  for  raters,—  The  Ratine?  of 



Personal  Traits  clearly  articulates  its  purpose.  There 
are  instructions  for  its  use  and  a fairly  adequate  defini- 
tion of  terms.  Little  information  regarding  the  method 
of  devising  the  scale  is  given.  No  material  on  the  subject 
of  reliability  and  validity  is  presented.  Instructions 
for  using  this  particular  scale  are  limited.  The  purpose 
of  the  instrument  is  well  stated  although  further  infor- 
mation should  have  been  included. 

2/ 

The  BEC  Personality  Rating  Schedule  does  not  men- 
tion the  background  and  need  for  rating  scales.  The 
purpose  of  this  particular  rating  scale  is  described 
but  not  in  sufficient  detail.  Methods  of  procedure  in 
devising  the  scale  are  omitted  as  well  as  the  subject 
of  reliability  and  validity.  The  two-page  instructions 
to  teachers  stress  the  administrative  routine  of  rating 
but  omit  sufficient  directions  for  the  Judges. 

1/ 

Better  rating  manuals. — My  Personality  Growth  Book 

T7T7  M.  Miller,  Rating  of  Personal  Traits.  The  Chron- 
icle. Fort  Byron,  New  York,  1941,  2 pp. 

2 / P.  J.  Rulon,  E.  A.  Nash,  and  G-.  Woodward,  et  al. 
Business  Education  Council  Personality  Rating  Schedule. 
Harvard  University  Press,  Cambridge,  1936,  2 pp. 

2/  W.  A.  McCall  and  J.  P.  Herring,  My  Personality  G-rowth 
Book.  Teachers  College,  Columbia  University.  New  York, 
1941,  12  pp. 
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gives  a brief  description  of  rating  scales  and  their 
need.  More  information  should  have  been  included,  but 
many  facts  have  been  considered.  The  purpose  of  this 
particular  rating  scale  is  adequately  outlined.  The 
procedure  of  devising  the  scale  should  have  been  pre- 
sented with  more  significant  detail.  Reliability  and 
validity  coefficients  are  reported  in  the  manual.  There 
are  detailed  instructions  for  using  the  scale,  but  con- 
siderable improvement  could  be  directed  toward  trait  de- 
finitions . 

The  manual  of  directions  for  the  Hags’erty-Olson- 

1/ 

Wlckman  Behavior  Schedules  refers  slightly  to  the  back- 
ground of  rating  scales  but  adequately  covers  the  need 
for  such  instruments.  The  purpose  of  this  particular 
rating  scale  is  fairly  well  outlined.  Detailed  infor- 
mation regarding  the  manner  in  which  the  scale  was  de- 
vised is  presented.  There  is  an  adequate  discussion  of 
the  reliability  and  validity  of  the  rating  schedule. 
Instructions  for  using  the  scale  are  indicated.  Fur- 
there  detailed  definition  of  traits  is  needed. 

Refinements  of  A Satisfactory  Manual 

Points  for  inclusion. — Information  gathered  from 

speaking  and  writing  to  investigators  as  well  as  reading 

l/  M.  E.  Haggerty,  W.  C.  Olson,  and  E.  K.  Wickman, 
Haggerty-Olson-Wlckman  Beha/ior  Rating  Schedules.  World 
Book  Co.  fonkers,  New  f0rk,  19'BO,  b pp. 
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the  literature  suggests  that  a rating  scale  manual 
should  include  the  five  points  outlined  below.  These 

1/ 

factors  have  also  been  considered  necessary  by  Tiegs 
in  his  discussion  of  the  importance  of  an  adequate 
manual  in  the  selection  of  tests. 

(1)  A brief  description  of  the  background  of  ra- 
ting scales  and  the  need  for  them  should  be  furnished. 
Many  scales  in  industry  and  school  have  neglected  even 
a preliminary  statement  regarding  the  background  of  ra- 
ting scales.  The  need  for  rating  scales  has  been  ta- 
ken for  granted,  but  little  information  has  been  given 
raters . 

(2)  The  purpose  of  the  particular  rating  scale 
should  be  outlined.  Rating  scales  used  in  the  schools 
have  been  more  specific  concerning  the  purpose  of  a 
particular  scale  than  those  in  industry.  One  questions 
how  a scale  can  accomplish  the  generalized  goals  attri- 
buted to  many  of  Industry’s  rating  scales.  The  pur- 
pose of  most  scales  is  far  too  loosely  defined  and  de- 
scribed . 

(3)  Information  regarding  the  manner  in  which  the 

scale  was  devised  should  be  presented.  Scales  employed 

l/  E.  W.  Tiegs,  Tests  and  Measurements  for  Teachers. 
Houghton  Mifflin  Company.  Boston,  1931,  pp.  2&6-2§7 . 
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In  the  schools  are  more  apt  to  tell  how  they  were  devis- 
ed than  those  in  industry.  Even  in  the  school  situation 
some  scales  lack  such  information. 

(4)  A discussion  of  reliability  and  validity  should 
be  given.  The  scales  used  in  the  schools  have  broached 
the  subject  more  often  than  those  in  industry.  A lack 
of  statistical  evidence  also  exists  in  the  school  rating 
scales.  With  the  exception  of  a very  few  large  concerns, 
industry  has  not  been  too  perturbed  about  reliability 
and  validity. 

(5)  Instructions  for  using  the  particular  scale 
and  adequate  definition  of  terms  should  be  presented. 

Both  school  and  industry  have  neglected  this  point  more 
than  any  other.  Terms  have  not  been  defined  in  many 
rating  scales,  and  in  others  definitions  are  ambiguous. 
Instructions  for  using  scales  have  been  those  needed 
for  administrative  purposes  in  the  broadest  sense.  They 
have  not  been  a real  aid  to  the  individual  rater  and 
have  been  far  too  inadequate.  Since  people  have  always 
rated  other  people,  it  has  been  fallaciously  assumed 
that  anyone  can  use  a rating  scale  without  any  real  pre- 
paration. In  industry  great  exactness  is  devoted  exclu- 
sively to  the  rating  records  and  formal  administration 
of  the  rating  program.  Little  time  is  spent  with  the 
men  who  do  the  rating  and  on  whose  results  time  and  money 
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is  expended,  A pronounced  lack  of  rating  instructions 
for  teachers  is  also  evident. 

t 

Inadequate  directions. — Constructive  manuals  accom- 
pany only  a few  rating  scales.  The  scale  itself  may 
conform  to  recognized  techniques.  Without  adequate  in- 
structions, however,  the  instrument^  value  is  sharply 
decreased.  It  is  surprising  how  few  investigators  have 
stressed  instructions  to  raters.  In  the  field  of  men- 
tal testing,  directions  must  be  followed  to  the  letter 
if  accurate  results  are  to  be  obtained.  Group  intelli- 
gence and  achievement  tests  are  far  advanced  in  compar- 
ison with  the  frontier  measurement  of  personality  tests. 
The  former  contain  a great  deal  of  explanation  and  in- 
struction that  must  be  followed  accurately  if  reliable 
results  are  expected.  In  contrast  many  instructions 
that  accompany  rating  scales  cover  only  one  meagre  page. 

The  need  for  better  rating  manuals  in  both  school 
and  industry  is  indicated.  A few  steps  are  gradually 
being  taken  in  this  direction.  Projects  of  this  type 
will  probably  lead  to  greater  efficiency  and  use  of  rat- 
ing scales. 
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CHAPTER  IV 


STATEMENT  OF  PROCEDURES 
Construction  of  the  Scale 

Various  types  of  rating  scales  have  been  described 
in  previous  Chapters,  and  the  method  of  devising  the 
Vocational  Service  Rating  Scale  is  outlined  in  the  Manual. 
In  this  Chapter  procedures  for  developing  the  Scale  are 
presented  in  more  detail. 

Selection  of  traits. — The  problem  resolved  itself 
into  two  primary  questions.  (1)  What  traits  should  be 
included  in  the  Vocational  Service  Rating  Scale.  (2) 

What  techniques  should  be  introduced  in  order  to  obtain 
the  advantages  of  previous  rating  scales  and  at  the  same 
time  eliminate  or  modify  many  of  their  weaknesses. 

Traits  included  in  the  Scale  were  based  upon  the 
opinions  of  475  employers  and  20  persons  in  the  field 
of  guidance  and  personnel  in  the  Greater  Boston  area. 
Personality  qualifications  for  job  openings  and  reasons 
for  discharge  were  given  by  the  employers.  The  informa- 
tion was  written  on  a job  specification  blank  used  by  a 
social  agency  devoted  to  educational  and  vocational  guid- 
ance and  placement.  The  guidance  counselors  submitted 
their  opinions  regarding  personality  characteristics 
highly  necessary  for  most  Jobs.  The  employers,  who  hire 
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unskilled  to  professional  help,  were  chosen  at  random 
from  the  files  of  a social  agency  devoted  to  education- 
al and  vocational  guidance  and  placement  activities. 

The  traits  incorporated  in  the  Scale  were  mention- 
ed at  least  twenty-five  times  by  employers,  at  least 
twice  by  personnel  workers,  and  twice  in  industrial  scales. 
The  475  employers  described  83  traits.  Each  trait  was 
listed  at  least  once,  and  18  were  noted  25  or  more  times. 
These  18  traits  were  presented  to  20  counselors  for  their 
opinions.  In  order  to  be  included  in  the  Scale,  each 
trait  had  to  be  considered  necessary  by  two  or  more  coun- 
selors. Fifteen  traits  met  the  selected  criterion  where- 
upon it  was  necessary  to  provide  descriptions  of  behav- 
ior for  each  trait. 

Definition  of  traits.—  The  name  for  each  trait 
required  careful  editing  and  interpretation  of  what  em- 
ployers said  or  implied.  One  of  the  difficulties  en- 
countered was  the  fact  that  employers  on  their  job  or- 
der qualifications  were  actually  considering  the  same 
trait  and  calling  it  by  another  name.  The  terms  relia- 
bility and  dependability,  for  example,  were  often  in- 
terchanged. Some  employers  described  behavior  qualifi- 
cations that  obviously  required  great  exactness  without 
calling  it  by  the  name  accuracy.  Facts  such  as  these 
made  naming  and  defining  of  terms  an  essential  feature 
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of  the  rating  manual.  A detailed  definition  of  traits 
appears  in  the  Vocational  Service  Rating  Scale  Manual. 
Traits  were  defined  with  the  aid  of  Webster1 s New  In- 

17 

ternatlonal  Dictionary  of  the  English  Language  and 
Warren’ s Dictionary  of  Psychology. 

Grading  of  scale.--  The  decision  as  to  the  optimal 
number  of  steps  in  any  scale  is  largely  an  empirical 
matter.  If  too  few  steps  are  employed,  the  scale  is 
obviously  a coarse  one,  and  much  of  the  efficiency  of 
the  raters'  judicial  powers  is  lost.  Champney’s  and 

, 2/ 

Marshall’s  research  showed  that  in  scoring  two  graphic 
rating  scales,  judgments  were  more  consistently  record- 
ed on  the  finer  scale. 

On  the  other  hand,  the  scale  may  be  graded  so  fine- 
ly that  it  is  beyond  the  raters’  powers  to  dlscrimin- 

V 

ate  between  one  step  and  the  next.  Fryer  maintains 

there  can  not  be  more  than  five  degrees  of  proficiency 

for  an  objectively  defined  trait  in  a rating  scale. 

1/  Webster's  New  International  Dictionary  of  the  Eng- 
lish Language.  Unabridged.  G.  & C.  Merriam  Company. 
Springfield,  Massachusetts.  194-1,  xcvi  3210  pp. 

2/  H.  C.  Warren,  Dictionary  of  Psychology.  Houghton 
Mifflin.  Boston,  1934,  x 372  pp. 

3 / H.  Champney  and  H.  Marshall,  "Optimal  Refinement 
of  the  Rating  Scale",  Journal  of  Applied  Psychology: 
1939:  23:  323-331. 

4/  D.  Fryer,  "Rating  A Rating  Scale",  Industrial  Man- 
agement : 1927:  63:  301. 
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The  fineness  will  also  depend  upon  the  willingness  of 
the  Judge  to  make  the  effort  required  for  fine  discrim- 

1/ 

inatlon.  Conklin  concluded  after  an  exhaustive  analy- 
sis of  some  23,000  Judgments,  that  for  untrained  raters 
the  maximum  number  of  steps  should  be  five  for  a single 
scale.  A single  scale  is  defined  as  one  which  extends 
from  zero  to  the  maximum*  Most  rating  scales  in  school 
and  industry  have  used  five  degrees  of  proficiency. 

Description  of  behavior  for  each  degree  of  trait.— 
Trait  degrees  were  developed  from  behavior  descriptions 
given  by  employers  on  Job  orders,  examination  of  rating 
scales  in  school  and  industry,  opinions  of  personnel 
workers,  and  suggestions  from  articles  on  rating  scales. 
These  statements  were  all  carefully  edited  in  order  to 
convey  the  same  meaning  to  different  persons.  Each 
statement  was  typed  on  a separate  slip  of  paper.  Ten 
personnel  workers  were  asked  to  sort  the  slips  into 
five  piles  representing  degrees  of  success  for  each 
trait,  ranging  from  the  minimum  rating  (pile  one)  to 
the  maximum  rating  (pile  five).  After  editing  and  sort- 
ing was  completed,  the  only  descriptions  that  were  ac- 
cepted were  those  on  which  there  was  agreement  by  the 
Judges.  Agreement  was  interpreted  as  meaning  that  an 
item  must  fall  within  an  adjacent  pile  by  the  different 

l/  E.  S.  Conklin,  "The  Scale  of  Values  Method  for  Studies 
in  Genetic  Psychology",  University  of  Oregon  Publication: 
1923:  2:  1-36. 
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judges  in  order  to  be  included*  The  pile  at  which  the 
description  was  finally  included  was  the  one  with  the 
highest  frequency. 

Rules  for  the  construction  of  a graphic  rating 
scale . — Various  rules  are  mentioned  in  the  literature 
as  most  advisable  in  the  construction  of  a graphic  rat- 
ing scale.  In  general  many  of  these  rules  are  disre- 
garded. Inasmuch  as  the  procedure  for  constructing 
the  Vocational  Service  Rating  Scale  was  to  follow  them, 
they  are  listed  in  detail. 

(1)  Each  trait  should  occupy  a page  by  itself  or 
instructions  should  provide  that  all  individuals  be 
rated  for  the  same  trait  before  rating  the  other  traits 

(2)  The  line  should  be  at  least  five  inches  long 
but- not  too  long  so  that  it  may  be  easily  grasped  as 
a whole. 

(3)  The  descriptive  phrases  should  be  in  small 
type  with  considerable  white  space  between  them. 

(4)  Three  or  five  descriptive  adjectives,  two  ex- 
tremes and  one  or  three  intermediates,  should  be  employ 
ed* 

(5)  The  end  phrases  should  not  be  so  extreme  in 
meaning  that  raters  avoid  them. 

(6)  The  average  or  neutral  phrase  should  be  at 
the  center  of  the  line. 
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(7)  Descriptive  phrases  do  not  need  to  be  evenly 
spaced.  The  meaning  of  the  intermediate  ones  should 
be  nearer  the  middle  phrase  than  the  extreme  ones. 

(8)  A stencil  that  divides  each  line  into  several 
sections  to  which  numerical  values  are  assigned  should 
be  used  in  scoring. 

(9)  Any  finer  discriminations  in  rating  than  are 
used  in  scoring  should  not  be  required. 

Use  of  the  Vocational  Service  Rating  Scal6.—  Upon 
completion  of  the  first  form  of  the  Scale,  it  was  used 
in  guidance  and  placement  at  a social  agency  in  order 
to  discover  from  practical  experience  what  revisions 
might  be  necessary.  All  clients  other  than  "spot"  place- 
ments were  rated  by  the  counselors. 

About  this  time  the  Scale  was  also  introduced  in- 
to a settlement  house  where  six  Judges  rated  27  club 
members  whom  they  knew  in  common.  The  Pearsonlan  Pro- 
duct Moment  correlation  between  the  average  ratings 
of  Judges  A,  B,  C,  and  D,  E,  F,  were  taken  for  the  15 
traits  comprising  the  Scale.  Of  the  six  Judges,  none 
of  the  raters  were  "professional"  workers.  A manual 
had  not  been  written,  but  rules  of  rating  were  discuss- 
ed orally  in  detail  before  rating  took  place.  Although 
conditions  were  not  favorable,  the  try-out  of  the  test 
was  helpful  for  future  improvements.  Results  ranged 
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from  a correlation  of  ,41  in  self-confidence  to  a cor- 
relation of  .88  in  physical  vitality. 

Revised  edition  of  the  Vocational  Service  Rating 
Scale.—  The  revised  edition  of  the  Scale  is  an  attempt 
to  remedy  mistakes  and  provide  improvements  that  were 
not  apparent  until  the  Scale  had  been  used  extensively. 
It  is  not  radically  different  from  the  original  edition. 
A Manual  of  Instructions  was  written  for  the  revised 
Scale.  Instead  of  requiring  a scoring  stencil,  the 
revised  Scale  was  so  devised  that  scores  could  be  read 
directly.  A better  grade  of  paper  and  type  was  employ- 
ed, and  certain  minor  physical  changes  in  the  appear- 
ance of  the  Scale  were  made.  Additions  and  omissions 
of  behavior  description  for  each  degree  of  trait  were 
carried  out  in  the  same  manner  as  in  the  first  form 
of  the  Scale.  Changes  were  evolved  through  practical 
use  of  the  test. 

Upon  completion  of  the  revised  Scale,  it  was  tried 
in  other  situations  to  see  whether  the  Scale  was  mea- 
suring something  consistent  in  the  behavior  of  various 
groups.  There  were  25  subjects  from  the  Greater  Bos- 
ton area  in  each  of  the  four  groups.  The  groups  were 
as  follows:  (1)  high  school  boys  all  members  of  a typ- 
ical club  in  a large  neighborhood  house;  (2)  girls  in 
their  senior  year  of  a high  school  with  approximately 
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6 00  pupils;  (3)  men  all  performing  semi-skilled  work 
in  a company  that  manufactures  plastic  specialties  and 
employs  about  1,500  personnel;  and  (4)  student  nurses 
in  their  third  year  of  training  at  a 250-bed  hospital* 

Development  of  the  Manual 

Points  for  the  construction  of  a rating  scale  man- 
ual*— In  the  preceding  Chapter  points  for  inclusion 
in  a satisfactory  rating  scale  manual  are  discussed* 
Inasmuch  as  the  procedure  for  developing  the  Vocational 
Service  Rating  Scale  Manual  was  to  follow  these  points, 
they  are  summarized  briefly:  (1)  a brief  description 
of  the  background  of  rating  scales  and  their  need;  (2) 
the  purpose  of  the  Vocational  Service  Rating  Scale; 

(3)  the  method  of  devising  the  Scale;  (4)  reliability 
and  validity  of  the  Scale;  and  (5)  instructions  for  the 
use  of  the  Scale  and  definition  of  terms.  Although 
for  purposes  of  discussion  this  Chapter  is  divided  into 
three  main  sections,  the  procedures  for  developing  each 
one  of  them  are  interrelated  and  dependent  upon  one 
another. 

Manual  of  Instructions  for  the  Vocational  Service 
Rating  Scale. — A preliminary  introduction  to  the  Man- 
ual emphasizes  that  there  can  be  no  worthwhile  purpose 
to  rating  without  completely  reading  and  following  the 
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Instructions.  Many  rating  scales  have  brief  directions 
on  the  scale  itself,  which  is  a disadvantage.  Rating 
scales  are  extremely  interesting.  The  most  ardent  cri- 
tics will  accept  this  premise.  If  instructions  are 
written  on  the  scale  itself,  the  average  judge  more  of- 
ten than  not  glances  at  or  neglects  to  read  them  before 
starting  to  rate.  Rating  looks  easy,  and  there  is  the 
temptation  to  begin  at  once.  Accurate  ratings  take 
time  • 

The  Manual  is  an  attempt  to  present  significant 
material  condensed  in  a manner  interesting  both  to  the 
layman  and  the  trained  personnel  worker.  The  Manual 
is  written  for  the  rater  upon  whom  the  success  of  any 
rating  scale  essentially  depends.  If  a trained  person 
is  available,  the  Manual  with  its  accompanying  selected 
bibliography  may  be  used  as  a text  with  as  much  further 
discussion  prior  to  rating  as  is  deemed  advisable. 

The  Manual  of  Instructions  for  the  Vocational  Ser- 
vice Rating  Scale  is  presented  in  the  Appendix. 

Selection  and  Training  of  Raters 

1/  ~ 

Number  of  judges.--  Rugg  advocates  that  one  should 

use  the  pooled  ratings  of  not  less  than  three  Judges. 

I T H.  0.  Rugg,  "Self-Improvement  of  Teachers  Through 
Self -Ratings" , Elementary  School  Journal;  1920:  20: 
670-684. 


. 

. . 


r\  ' . r 

• ’ ^ * - - 

> ' 

. 

i ) f . o . • ‘ ' .*  M i p / .t 

gfKjfc  c • 

. 

f . - ' -i  c ■ ir  > 

, ■ ’ ■ r l ' o >. 

' 

. !.  * 

' 

'lerrrtjsfa  4 ni  fieertsl'.rtco 

:•: r r ■::  r:  " ' -" 

y 1 ‘ r of  - $ ■ 0 • O'  u ' rir  ' r aJ  ’ *iw  I 

* 1 . . ' ■'•• 

. 

' 

- ‘ ' 

. 


p<  - *■  ->  ■ ‘ > ■ -•  <-'(  > ' f r *>«: 

\ - 


_ _ 

. • ' 

r r'  . '■-r  .v  - . . 


Moore  also  recommends  ratings  of  three  Judges  if  it  is 
possible  to  obtain  that  number*  Reliability  and  valid- 
ity of  ratings  shows  an  increase  with  the  number  of 
judges  employed. 

In  practice  it  is  extremely  difficult  to  obtain 
three  persons  who  know  in  common  a large  group  of  in- 
dividuals well  enough  to  rate  them  adequately  on  per- 
sonality traits.  This  has  been  a noticeable  handicap 
of  ratings  in  school  and  industry.  A few  large  indus- 
trial concerns,  however,  have  three  Judges. 

Selection  of  Judges. — Three  different  Judges  ra- 
ted 25  individuals  in  each  of  the  four  groups.  Although 
this  does  not  appear  to  be  too  large  a number  of  sub- 
jects, it  was  not  possible  to  find  three  people  who  knew 
more  than  25  people  in  common  well  enough  to  give  jus- 
tifiable ratings  and  not  superficial  guesses.  There 
was  also  the  added  necessity  of  obtaining  judges  who 

were  interested  in  oersonality  rating  and  who  had  sim- 

2/ 

liar  backgrounds.  Conrad  points  out  that  judges  do 

much  better  if  they  are  interested  in  the  ratings  they 

v B.  V.  Moore,  "Personnel  Selection  of  Graduate  Engi- 
neers", Psychological  Monograph:  1922:  30:  1-85. 

2/  H.  S.  Conrad,  "The  Bogey  of  the  'Personal  Equation' 
in  Ratings  of  Intelligence",  Journal  of  Educational 
Psychology : 1932:  23?  147-149,  and  "The  Personal  Equa- 
tions  in  Ratings:  An  Experimental  Determination",  Jour- 
nal of  Genetic  Psychology:  1932:  41:  267-293. 
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make  and  have  comparable  educational  and  professional 
backgrounds.  In  comparison  with  the  ratees,  the  judges 
should  also  be  on  a superior  occupational  level  in  the 
type  of  work  the  ratees  perform.  This  was  a further 
difficulty  in  securing  judges.  In  order  to  obtain  the 
best  possible  results  those  who  actually  or  supposedly 
have  reached  their  position  by  ability  and  experience 
should  theoretically,  at  least,  be  more  competent  to 
judge  others  in  their  own  field.  This  is,  of  course, 
debatable,  and  a great  deal  may  be  said  pro  and  con. 

The  three  judges  for  each  group  were  carefully 
selected  and  were  (1)  group  workers,  (2)  high  school 
teachers,  (3)  personnel  supervisors,  and  (4)  nurse  su- 
pervisors. The  writer  contacted  the  person  who  was  to 
be  chairman  of  each  of  the  four  groups.  Each  chairman 
had  at  least  a Bachelor* s degree,  and  two  had  three 
or  more  courses  in  guidance.  With  the  exception  of  the 
club  situation  with  two  volunteers,  all  the  Judges  were 
professional  workers  and  had  at  least  five  years  of 
experience  in  their  respective  fields.  All  the  raters 
expressed  Job  satisfaction  in  their  own  work  and  an 
active  interest  in  knowing  and  helping  the  individuals 
whom  they  supervised.  They  all  mentioned  an  interest 
in  rating  scales  prior  to  their  participation  in  this 
study.  The  attitude  of  the  Judges  was  most  favorable. 
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The  raters  regarded  their  active  participation  as  an 
opportunity  for  self-growth  rather  than  an  additional 
duty. 

Training  of  .judges. — After  reading  the  literature 
on  rating  scales  attention  is  focussed  on  the  fact  that 
a disproportionate  amount  of  attention  is  given  to  the 
construction  and  description  of  rating  scales  and  rel- 
atively little  to  the  judges.  The  inadequacy  of  in- 
structions for  judges  is  a major  cause  of  weakness  in 

1/ 

many  rating  scales.  Driver  remarks  that  90  per  cent 
of  the  difficulties  of  rating  may  be  attributed  to  the 
judges  and  only  10  per  cent  to  the  rating  form. 

The  Scales  and  Manuals  were  sent  to  each  chairman 
through  the  mails.  All  chairmen  were  informed  that 
ratings  were  to  be  completed  independently.  Before 
arrival  of  the  Vocational  Service  Rating  Scale  each  set 
of  three  judges  had  a preliminary  conference  to  select 
the  25  individuals  for  their  respective  group.  No  dis- 
cussion concerning  ratings  took  place  at  this  meeting, 
but  care  was  taken  to  make  certain  that  the  individuals 
chosen  were  well  known  to  each  member  of  the  committee. 
The  Judges  were  given  the  manuals  and  read  them  before 
they  received  the  blanks  for  rating  the  25  subjects. 

1/  R.  S.  Driver,  "A  Case  History  in  Merit  Rating”,  Per- 
sonnel: 1940:  16:  150. 


• •• 


- 

. • : 

:j  z>rf:ut  * c.  w 1 .*  i n*t- 

. 


• • •••-.:  c ' . t •- 

■ ■ ■>:  r •'  'J  ■ •'  * *tc  ' vJhtuJB 

■ 

-•/  • ; .J  - 'v  ‘ : 

. 

. 

. 


» • 


j -i  er 


^ . 

: 


: lennc e 


Summary 


This  Chapter  covers  the  general  procedures  of 
the  investigation.  The  three  major  topics  are  construc- 
tion of  the  Scale,  development  of  the  Manual,  and  se- 
lection and  training  of  the  raters.  Sufficient  infor- 
mation is  given  in  order  that  a similar  study  may  be 
developed.  Detailed  analysis  and  interpretation  of 
the  specific  statistical  techniques  are  presented  in 
Chapter  V* 
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CHAPTER  V 


STATISTICAL  RESULTS  OF  THE  VOCATIONAL  SERVICE  RATING  SCALE 

Before  any  measuring  instrument  may  be  considered 
worthwhile,  it  must  be  shown  to  be  reliable  and  valid. 

The  methods  followed  in  pointing  out  the  validity  and 
reliability  of  the  Vocational  Service  Rating  Scale  are 
those  of  quantitative  and  qualitative  approaches. 

This  Chapter  deals  with  the  statistical  method;  Chapter 
VI  covers  the  qualitative  approach. 

Reliability  and  Validity 
Importance  of  reliability  and  validity . — In 
the  discussion  of  reliability  and  validity,  authori- 
ties differ  in  regard  to  the  relative  importance  of 

1/ 

each.  Some  authors  like  Symonds  consider  reliability 

2/ 

to  be  more  important  while  Thurstons  says  that  no 

valid  instrument  can  have  a low  reliability. 

l/  M.  M.  Lombardi,  The  Inter-Trait  Rating  Technique. 
Teachers  College,  Columbia  University.  New  York. 
Contributions  to  Education  no.  760,  1938,  p.  49. 


2/  L.  L.  Thurstone,  The  Reliability  and  Validity  of 
Tests.  Edwards  Bros.  Ann  Arbor,  Michigan,  1932, 

p.  102. 
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Validity  of  the  test  consists  now  in  their 
consistency  with  the  conventional  expressions 
of  the  trait  and  since  reliability  also  refers 
to  their  consistency,  we  see  that  fundamentally 
the  two  concepts  are  identical.  Their  only 
practical  difference  is  that  validity  refers  to 
the  consistency  of  an  index  that  is  generally 
accepted  and  one  that  is  new  or  strange,  while 
reliability  refers  to  the  consistency  of  an 
index  with  another  Just  like  it.  1/ 

Definition  of  reliability  and  validity  as  applied 

to  the  Scale.—  The  term  "reliability"  may  be  defined 

as  the  consistency  with  which  the  Vocational  Service 

Rating  Scale  measures  whatever  it  does  measure. 

Many  viewpoints  concerning  methods  of  validating 

tests  or  scales  have  been  presented  in  the  literature. 

For  the  most  part  investigators  do  not  agree  upon  the 

various  methods  that  may  be  employed.  Because  of  the 

difficult  nature  of  the  problem,  some  investigators 

believe  that  the  particular  test  or  scale  should  be 

subjected  to  more  than  one  method  of  validation. 

On  the  basis  of  his  own  experience,  the  writer 

favors  the  viewpoint  of  Investigators  who  have  had 

both  theoretical  training  in  psychology  as  well  as  va- 

2/ 

rled  practical  experience.  McQuitty  supports  this 
contention  when  he  points  out  that  many  industrial 
1/  Ibid. 

2/  L.  L.  McQuitty,  "Developing  Applied  Psychologists", 
The  American  Psychologist:  194-8:  3:  16-19. 
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psychologists  are  much  less  effective  than  they  might 
be,  because  they  are  not  well  grounded  in  the  non-psy- 
chological  aspects  of  the  fields  in  which  they  apply 
their  science. 

Remmers  has  a wealth  of  theoretical  and  practical 
training.  In  discussing  the  Purdue  Rating  Scale  for 

1/ 

Instructors . on  which  students  rate  teachers  on  ten 
traits,  he  considers  reliability  and  validity  to  be 
synonymous . 

...While  reliability  may  be  defined  as 
the  accuracy  with  which  a measuring  instru- 
ment measures  whatever  it  does  measure,  va- 
lidity is  defined  as  the  extent  to  which  the 
instrument  measures  what  it  purports  to  mea- 
sure. Since  it  is  student  judgments  that  con- 
stitute the  criterion,  reliability  and  validity 
are  in  this  case  synonymous.... 

Since  it  is  the  judgment  of  employers  and  person- 
nel counselors  that  constitutes  the  criterion  for  the 
Vocational  Service  Rating  Scale,  reliability  and  va- 
lidity are  in  this  case  synonymous.  If  employers  and 
counselors  regard  these  traits  as  essential  for  voca- 
tional adjustment  and  if  these  traits  are  consistently 
rated  by  different  judges  in  different  situations,  the 
Scale  is  valid. 

l/  H.  H.  Remmers,  "Reliability  and  Halo  Effect  of  High 
School  and  College  Students'  Judgment  of  Their  Teachers" 
Journal  of  Applied  Psychology:  1934- : 18:  621. 
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Other  evidence  of  the  validity  of  the  Scale  will, 
nevertheless,  be  presented.  The  problem  of  validity  in 
personality  measurement  has  not  been  conclusively  settled. 
Presentation  of  additional  methods  of  determining  the 
validity  of  the  Scale  appears  necessary  in  a field  as 
controversial  as  that  of  rating  scales. 

Statistical  Procedure 

The  method  employed  to  show  reliability  and  validity 
of  the  Scale  is  to  compare  the  ratings  of  three  judges 
who  rated  25  individuals  on  15  traits.  There  are  four 
situations,  100  subjects  (25  in  each  group)  and  12  judges 
(three  in  each  situation).  Tables  9 to  20  in  the  Appen- 
dix give  the  raw  scores  for  each  judge’s  rating  of  25 
subjects  on  each  of  the  15  traits.  Table  21,  the  Appen- 
dix, lists  the  arithmetic  mean  for  each  judge's  rating 
of  25  individuals  on  each  of  15  traits.  Table  22  of  the 
Appendix  shows  the  standard  deviation  of  each  trait  for 
each  judge.  These  tables  are  presented  for  the  con- 
venience and  information  of  the  readers  in  order  that 
they  may  see  the  source  from  which  the  writer  treated 
the  raw  statistical  data. 

Coefficients  of  correlation.--  Table  3 presents 
the  coefficients  of  correltion  between  the  ratings  of 


,■  • ' ' V 0 : 

. 

. a-  ■ ■ 


s-a 

. 

# 

. 

- ■ r 

- 

* 

. ■ SD  r ? V’  1 


three  Judges  In  each  group.  The  correlation  of  three 
original  ratings  may  be  taken  as  the  reliabilities  of 
the  three  Judgments,  This  Involves  the  same  principle 
as  taking  the  correlation  of  forms  A and  B of  a test 
as  the  reliabilities  of  the  forms.  The  average  coef- 
ficient of  correlation  was  computed  by  a method  orig- 

1/ 

inated  by  Fisher.  The  basic  formula  is 
~L  - '/^ 

in  which  z is  the  transformed  variable  which  for  almost 
any  population  correlation  is  normally  distributed,  logQ 
the  natural  logarithm,  and  r the  coefficient  of  corre- 
lation. For  each  of  the  three  coefficients  of  corre- 
lation to  be  averaged,  z is  obtained  directly  from  a 

1/ 

table  in  Fisher's  book.  The  mean  of  the  three  z's 
is  calculated.  From  the  same  table  the  r for  the  mean 
z is  found;  this  is  the  average  coefficient  of  corre- 
lation. Weighting  oj  z is  unnecessary  in  this  study 
because  the  samples  are  all  the  same  size. 

Correlations  ranged  from  a low  of  .6 2 on  trait 
15  for  boy's  club  to  a high  of  .96  on  trait  10  for  high 
school  students.  A few  tests  of  intelligence  and  achieve 
ment  have  reliability  coefficients  as  high  as  .95  to 
•98,  In  the  frontier  field  of  personality  measurement, 

1/  R.  A.  Fisher,  Statistical  Methods  for  Research  Workers 
Oliver  and  Boyd,  Ltd.  London,  1930,  3rd  edition,  pp.  16>3- 
177. 

2 / Ibid.,  p.  177. 


. 

* f + r;  ■ pof.  t ocf  r u;.- 'il'ic 

. , •'  ••  " v' 

- 

\jc 

. 

- [jalja  'lo'i  £ v ftecrcolen^it  »xi£  s 1 s rioJiriw  xxl 

^ • ' ‘ ■ ■ •'  - 

- ■ - ’ . i ••  ■; J it 1 : i 

— ■ ' 1:  ‘ ■ 

■ ■ • ' . . ■ '-*>*  ■ ' -r 

1 • » T - 

. 

" • # • 

. •.  • 'j  ■!  yJ 

no  1o  W 

. 1 ■ o*l 

. 

, 

t : » 

. * * . 

-TTI 


Table  3 

Coefficient  of  Correlation  and  Probable  Error 
Between  Judges  in  Each  Situation  for  Each  Trait 
(23  subjects  and  3 judges  for  each  situation) 


High  School 


Traits 

Boy' 

s Club 

Students 

Industry 

Nurses 

r 

PE 

r 

PE 

r 

PE 

r 

PE 

1 

.71 

• 066 

.85 

.038 

.85 

.038 

.79 

.051 

2 

.82 

.043 

.91 

.024 

.78 

.053 

.69 

.070 

3 

.75 

.059 

.85 

.038 

.79 

.051 

.66 

.076 

4 

.79 

.051 

.89 

.028 

.89 

.028 

.73 

.063 

5 

.69 

.070 

.93 

.018 

.82 

.043 

.71 

.066 

6 

.76 

.058 

.92 

.022 

.73 

.063 

.78 

.053 

7 

.79 

.051 

.95 

.013 

.87 

.034 

.82 

.043 

8 

.82 

.043 

.95 

.013 

.89 

.028 

.68 

.073 

9 

.66 

.076 

.94 

.016 

.77 

.054 

.70 

.069 

10 

.77 

.054 

.96 

.011 

.83 

.042 

.66 

.07  6 

11 

.77 

.054 

.95 

.013 

.81 

.046 

.66 

.076 

12 

.80 

.049 

.94 

.016 

.82 

.043 

.66' 

.076 

13 

.71 

•066 

.87 

.034 

.84 

.040 

.79 

.051 

14 

.83 

.042 

.93 

.018 

.85 

.038 

.88 

.031 

15 

.62 

.083 

.92 

.022 

.73 

.063 

.66 

.07  6 
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however,  as  pointed  out  in  Chapter  II,  reliability  and 
validity  coefficients  have  been  much  lower.  The  relia- 
bility and  validity  coefficients  of  the  Vocational  Ser- 
vice Rating  Scale  range  from  .62  to  .96  with  more  than 
half  at  .80  or  above. 

The  correlation  coefficient  for  the  three  judges 
in  the  high  school  and  industrial  situations  for  appear- 
ance is  .85.  The  probable  error  of  .038  means  that  the 
chances  are  even  (50  in  100)  that  the  true  coefficient 
of  correlation  (r)  falls  within  the  limits  of  .85  plus 
or  minus  .038  or  between  .89  and  .81  and  that  the  chan- 
ces are  99  in  100  that  the  true  r falls  within  the  lim- 
its of  .85  plus  or  minus  four  times  .038  or  between 
1.00  and  .70. 

To  take  another  example  of  the  meaning  of  Table  3, 
the  r for  the  three  judges  in  nursing  on  appearance  is 
•79  plus  or  minus  .050.  This  means  the  chances  are  50 
in  100  that  the  true  r falls  between  .74  and  .84  and 
they  are  99  in  100  that  the  true  r falls  between  .59 
and  . 99 • 

In  the  boy's  club  situation,  trait  2,  the  coeffi- 
cient of  correlation  is  .82  plus  or  minus  .043,  which 
means  that  there  are  50  chances  in  100  that  the  true  r 
falls  between  .78  and  .86  and  that  there  are  99  chances 
in  100  that  the  true  r falls  between  .65  and  .99. 
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In  the  high  school  and  industrial  groups,  trait  4, 
the  coefficient  of  correlation  is  ,89  plus  or  minus  *028. 
This  is  read  as  meaning  that  there  are  50  chances  in  a 
100  that  the  true  r falls  between  .92  and  .86  and  that 
there  are  99  chances  in  100  that  the  true  r falls  be- 
tween 1.00  and  .78. 

For  the  high  school  group  on  trait  11,  the  corre- 
lation is  .95  plus  or  minus  .013*  This  means  that  the 
true  r falls  within  the  limits  of  .93  and  .97  in  50 
chances  out  of  100  and  between  .90  and  1.00  in  99  chan- 
ces out  of  100. 

Spearman-Brown  Prophesy  Formula.--  The  ability 
of  the  Spearman-Brown  Prophesy  Formula  to  predict  in- 
creased reliability  of  ratings  for  an  increased  number 

i/ 

of  judges  has  been  shown  many  times*  The  Formula 
may  be  used  to  estimate  the  Increased  reliability  of 
the  Vocational  Service  Rating  Scale  if  there  are  four, 
six,  or  more  judges.  It  may  also  be  used  in  ratings 
to  determine  how  many  judges  are  needed  to  raise  an 
obtained  correlation  to  a certain  desired  optimum. 

Table  4 shows  the  estimated  coefficient  of  correla- 
tion when  the  number  of  judges  is  doubled,  i.e.,  in- 
creased from  three  to  six.  The  Spearman- Brown  Prophesy 

1/  E.  L.  Clark,  "Spearman- Brown  Formula  Applied  to  Rat- 
ings of  Personality  Traits" , Journal  of  Educational 
Psychology : 1935  s 26:  552-555* 
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Table  4 


Coefficient  of  Correlation  and  Probable  Error 
When  Number  of  Judges  is  Doubled 
Using  Spearman-Brown  Prophesy  Formula 
(25  subjects  for  each  situation; 
judges  increased  from  3 to  6) 


High  School 

Traits  Boy  *3  Club  Students Industry  Nurses 


r 

PE 

r 

PE 

r 

PE 

r 

PE 

1 

00 

• 

.046 

.92 

.022 

.92 

.022 

.88 

.032 

2 

.90 

.027 

.95 

.013 

.88 

.032 

.82 

.049 

3 

.86 

.039 

.92 

.022 

.88 

.032 

.80 

.055 

4 

.88 

.032 

.94 

.016 

.94 

.016 

.84 

.042 

5 

.82 

.049 

.96 

.010 

.90 

.027 

.83 

.046 

6 

.86 

.039 

.96 

.010 

.84 

.042 

.88 

.032 

7 

.88 

.032 

.97 

.004 

.93 

.018 

.90 

.027 

8 

.90 

.027 

.97 

.004 

.94 

.016 

.81 

.052 

9 

.80 

.055 

.97 

.004 

.87 

.035 

.82 

.049 

10 

.87 

.035 

.98 

.003 

.91 

.025 

.80 

.055 

11 

.87 

.035 

.97 

.004 

.90 

.027 

.80 

.055 

12 

.89 

.030 

.97 

.004 

.90 

.027 

.80 

.055 

13 

.83 

.046 

.93 

.018 

.91 

.025 

.88 

.032 

14 

.91 

.025 

.96 

.010 

.92 

.022 

.93 

.018 

15 

.76 

.064 

.96 

.010 

.84 

.042 

.80 

.055 

Formula  is  the  method  of  computation  employed.  Table  4 
is  read  in  the  same  manner  as  Table  3* 

When  the  number  of  judges  is  doubled,  the  lowest 
coefficient  of  correlation  is  *76.  Of  60  calculations, 

30  fall  at  .90  or  better,  and  29  from  .80  to  .89*  These 
results  lead  to  the  conclusion  that  if  the  judges  are 
comparable,  the  reliability  of  pooled  judgments  increases 
directly  with  the  number  of  Judges  according  to  the 
Spearman-Brown  Prophesy  Formula. 

In  most  instances  it  is  not  possible  to  obtain  six 
Judges  at  one  time  who  know  a large  enough  group  suffi- 
ciently well  to  rate  them  on  the  Vocational  Service 
Rating  Scale.  This  is  not  an  insurmountable  difficulty. 
The  Scale  shows  satisfactory  reliability  and  validity 
with  the  three  judges  to  be  considered  a good  instrument 
for  the  purpose  for  which  it  was  constructed.  If  six 
Judges  are  used,  the  reliability  and  validity  of  the 
Scale  is  sufficiently  increased  to  warrant  the  time  and 
effort  involved. 

Index  of  reliability.—  Another  method  of  showing 

validity  of  the  Scale  is  that  of  the  index  of  reliability 
which  gives  the  correlation  between  obtained  scores  and 
theoretically  true  scores  in  the  same  function.  The  in- 
dex is  in  reality  a validity  coefficient  since  it  tells 
how  well  a test  is  measuring  true  ability  in  the  function 
which  it  purports  to  measure.  The  correlation  between 
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a series  of  obtained  scores  and  their  corresponding  "theo- 

1/ 

retically  true"  scores  may  be  found  by  the  formula 
/b,  0*5-  s^/v,  X in  which  A]  X is  the  reliability  coefficient 
of  the  given  test  and  A\  represents  the  correlation  be- 
tween the  obtained  and  true  scores.  The  index  of  relia- 
bility gives  the  maximum  correlation  which  a particular 
test  is  capable  of  yielding.  The  symbol  ckd  is  used  to 
designate  true  scores,  i.e.,  those  scores  gotten  from  an 
indefinite  number  of  administrations  of  the  Scale  to  the 
same  group. 

In  Table  6,  60  validity  coefficients  (15  for  each 
group)  are  presented.  The  high-low  figures  for  each 
group  are  shown  in  Table  5. 

Table  5 

High  and  Low  Validity  Coefficients 
For  Each  Situation 

Group  Low  High 

Boy's  Club  .79  .91 

High  School  Students  .92  .98 

Industry  .85  •94- 

Nurses  .81  .94 

Of  the  60  calculations,  30  fall  at  .90  or  better  and  29 

from  .80  to  .89.  The  index  of  reliability  in  all  four 

situations  is  sufficiently  high  to  warrant  the  statement 

that  the  Vocational  Service  Rating  Scale  is  an  extremely 

good  estimate  of  the  function  it  is  trying  to  measure. 

l/  H.  E.  Garrett,  op.  cit., 


p.  319 


oeri$M  v.ntl'rc-  es'fior  *il  eito  jjn£  ceicoe  BenJtBJrfo  lo  eei^xee  b 

. 

alkane  1 jjfjJ  ee*xo©s  eintf 

. 

■ • j : - 

•■  . r ‘ . 

:;oio,i  " 'li.  >£iw  act-  I - -.CO  r ■••.  ‘ .: ‘ : .**  " v w y • : M 

. / '■  *'■'  • 

. . . • 

-»  ; 1 1.  c.  It  to  enoiij  ■ if  ■ ■ . Jo  n.-  ■■-  3.J.: 

♦ 

' 

■ • 

# 

■ ' )1  'V  I -1 


■ 

r ' ' • • -T- 

V:  r-  ■ • 


. 


. 

. 


. 

. 

. 


. 

. 

ir  *r  id  *oJ  i r.'ro.M  ,'U  ‘ ' 

at  r,i  "cP  -'nXdJiH  eoJhn;*'??.  lJBfroldB9oV  erW  dflrU' 

. 

. • ' . , . . ..  . ■ * .7  . 


Table  6 


Index  of  Reliability 

(25  subjects  and  3 judges  for  each  situation) 


Traits 

Boy's  Club 

High  School 
Students 

Industry 

Nurses 

1 

.84 

.92 

.92 

.89 

2 

.91 

.95 

.88 

.83 

3 

.87 

.92 

.89 

.81 

4 

.89 

.94 

.94 

.85 

5 

.83 

.96 

.91 

.84 

6 

.87 

.96 

.85 

.88 

7 

.89 

.97 

.93 

.91 

8 

.91 

.97 

.94 

.82 

9 

.81 

.97 

.88 

.84 

10 

.88 

.98 

.91 

.81 

11 

.88 

.97 

.90 

.81 

12 

.89 

.97 

.91 

.81 

13 

.84 

.93 

.92 

.89 

14 

.91 

• 96 

.92 

.94 

15 

.79 

.96 

.85 

.81 

Halo.—  A third  method  of  showing  validity  of  the 


Vocational  Service  Rating  Scale  is  to  demonstrate  that 
halo  does  not  prevent  trait  discrimination  by  the  Judges* 
There  is  not  any  objective  standard  of  measure  of  the 
numerical  values  necessary  in  order  to  indicate  the 

1/ 

presence  of  halo.  Johnson  in  a scholarly  treatise  on 

Judgment  states  that  the  evidence  for  halo  is  thin. 

2/ 

Driver,  in  discussing  halo,  describes  the  term 
as  it  is  usually  conceived;  '‘Thus,  it  has  been  observed 
that  employees  rated  high  in  one  trait  tend  to  receive 
higher  ratings  than  they  deserve  on  all  other  traits." 

The  converse  is  also  true.  If  halo  is  operating,  by 
definition  there  will  be  little  or  no  spread  between 
scores  of  the  various  traits  for  individual  subjects. 

The  range  of  possible  scores  for  each  trait  on  the 
Vocational  Service  Rating  Scale  runs  from  1 to  50.  The 
highest  and  lowest  scores  and  the  difference  between  them 
is  calculated  for  each  subject  in  each  group  and  is  pre- 
sented in  Tables  23  to  34-  in  the  Appendix. 

For  boy's  club,  Judge  A,  the  spread  between  the 
highest  and  lowest  scores  of  subject  1,  is  32  points, 
resulting  from  a high  score  of  42  on  trait  12  and  a low 
score  of  10  on  trait  3*  In  the  case  of  Judge  3 for 
subject  1,  boy's  club,  the  range  of  45  results  from  a 

l/  D.  M.  Johnson,  "A  Systematic  Treatment  of  Judgment", 
Psychological  Bulletin:  1945:  42:  193-224. 

2/  R.  S.  Driver,  "A  Case  History  in  Merit  Rating",  Person 
nel:  1940:  16:  155. 
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high  of  49  on  trait  12  and  a low  of  4 on  trait  3*  For 
subject  1,  judge  C of  boy's  club,  the  range  is  21  points, 
obtained  from  a high  of  34  on  trait  12  and  a low  of  13 
on  traits  2 and  3* 

For  high  school  students,  judge  A rated  subject  3 
with  a high  of  40  on  traits  1,  2,  and  4,  and  with  a low 
of  5 on  trait  10,  producing  a spread  of  35  points. 

Judge  B of  the  same  group  rated  subject  3,  40  on  traits 
1 and  4,  and  on  trait  10  a low  of  5*  This  created  a 
range  of  35  points.  Subject  3 of  the  same  group  also 
received  from  judge  C a spread  of  35  points,  resulting 
from  a high  of  40  on  trait  1 and  a low  of  5 on  trait  10. 

Judge  A in  industry  rated  individual  1 with  a high 
of  48  on  trait  9 and  a low  of  11  on  trait  10  for  a dif- 
ference of  37  points.  Judge  B of  the  same  group  rated 
the  same  individual  50  on  trait  9 and  15  on  trait  10, 
with  a spread  of  35  points.  Judge  C of  the  same  group 
rated  the  same  individual  a high  of  45  on  traits  8,  9, 
and  12,  and  a low  of  18  on  trait  10,  for  a range  of  27 
points . 

Taking  subject  22  of  the  nurses,  Judge  A rated 
with  a high  of  30  on  traits  5 and  6,  and  a low  of  5 
on  trait  12  for  a spread  of  25  points.  Judge  B rated 
the  same  subject  in  the  nursing  group  a high  of  30  on 
trait  4 and  a low  of  10  on  trait  13,  creating  a range 
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of  20  points.  Judge  C of  the  same  group  rated  indivi- 
dual 22  with  a high  of  35  on  traits  4,  6,  and  9,  and  a 
low  of  10  on  trait  13,  for  a spread  of  25  points. 

The  difference  between  high  and  low  scores  for 
each  subject  is  larsrer  than  aopears  uoon  cursory  exam- 

ti7 

ination.  Remmers  comments  are  appropriate. 

Some  correlation  would,  of  course,  be  ex- 
pected even  if  no  halo  effect  were  present,  since 
the  weight  of  all  evidence  favors  the  conclusion 
that  desirable  measurable  characteristics  are 
positively  correlated.  The  amount  of  true  psy- 
chological concommitance  in  the  occurrence  of 
traits  is,  however,  indeterminate.... 

Logically  a general  impression  is  caused  by  some 
trait  or  traits  in  combination  or  singly  of  the  indi- 
viduals under  consideration.  This  has  been  indicated 

2/ 

in  the  research  of  Asch  where  he  points  out  that  there 
are  central  as  well  as  peripheral  traits  of  personality. 
A general  impression  may  be  distorted  by  the  fact  that 
a particular  trait  or  trait  pattern  reminds  us  of  fa- 
vorable or  unfavorable  incidents  of  the  past  and  unknow- 
ingly inf luences the  efforts  of  consclentous  ratings. 

The  trait  or  traits,  moreover,  on  which  a subject 
is  rated  consistently  high  or  low  by  different  judges 

l/  H.  H.  Remmers,  "Reliability  and  Halo  Effect  of  High 
School  and  College  Students'  Judgments  of  Their  Teachers" 
Journal  of  Applied  Psychology:  1934:  18:  628. 

2/  S.  E.  Asch,  "Forming  Impressions  of  Personality", 
Journal  of  Abnormal  and  Social  Psychology:  1946:  41: 
258-290. 
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are  the  central  traits  that  create  an  impression.  This 

1/ 

has  been  shown  in  the  Investigation  of  Asch  and  in 
this  study  has  also  been  indicated  in  the  introspections 
of  the  judges.  In  their  attempts  to  identify  the  spe- 
cific Individuals  they  had  rated,  the  judges  mentioned 
that  they  had  been  guided  by  the  extreme  scores  of  each 
subject . 

Does  the  spread  occur  just  as  greatly  for  those  in- 
dividuals rated  very  high  on  particular  traits  as  for 
those  rated  particularly  low  and  as  for  those  who  were 
not  rated  either  particularly  high  or  low.  The  highest 
scores  for  all  Judges  in  all  four  situations  for  each 
of  the  15  traits  were  tabulated.  For  example,  judge  A, 
boy’s  club,  subject  9 received  the  highest  score  on  trait 
1.  Subject  9' s range  of  scores  on  all  traits  was  44-26. 
Thus,  individual  9 appears  under  the  heading,  Those 
Subjects  Who  Scored  Highest  on  at  Least  One  Trait,  in 
Table  35,  the  Appendix.  The  same  procedure  was  fol- 
lowed for  the  low  scores.  For  the  same  group  and 
Judge,  subject  17  received  the  lowest  score  on  trait 
1.  Individual  17' s trait  scores  ranged  from  a high  of 
19  to  a low  of  4.  Subject  17  appears  under  the  head- 
ing, Those  Subjects  Who  Scored  Lowest  on  at  Least 
One  Trait,  in  Table  35.  After  all  the  subjects  who 
1/  Ibid. 
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had  scored  highest  and  lowest  on  at  least  one  trait  were 
tabulated,  the  remaining  individuals  were  listed  and  the 
range  of  their  scores  Included  in  Table  35  under  subtitle. 
Other  Scores.  This  data  is  presented  in  Tables  35  to 
46  of  the  Appendix. 

The  spread  between  high-low  scores  is  as  great 
or  greater  in  most  instances  for  the  highest  and  lowest 
rated  subjects  on  any  trait  as  is  the  spread  for  those 
not  rated  highest  or  lowest  on  any  trait.  If  halo  were 
operating,  one  would  expect  that  the  spread  of  the  high- 
est-lowest groups  would  be  less  than  for  the  others. 

For  all  judges  in  boy’s  club,  industry,  and  Judges  A 
and  B of  the  nurses,  the  results  show  a wider  range 
for  the  extreme  ratees  than  for  the  others.  In  the 
high  school  situation  the  highest  rated  group  has  the 
smallest  spread,  the  lowest  rated  group  the  widest  range, 
and  the  others  have  a spread  between  that  of  the  highest 
and  lowest  rated  groups.  If  halo  were  operating,  the 
subjects  receiving  the  extreme  scores  on  any  trait  should 
be  expected  to  have  less  of  a spread  between  their  in- 
dividual high  and  low  scores  than  other  subjects.  Tables 
35  to  46  show  that  this  is  not  the  case. 

Many  investigators  believe  that  if  there  is  a wide 
difference  between  the  rating  which  a subject  receives 
on  one  trait  and  the  one  he  receives  on  another,  the 
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ratings  are  likely  to  be  accurate  pictures  of  the  indi- 
vidual. This  point  of  view  is  widely  held.  Every  trait 
is  scored  highest  or  lowest  at  least  once  for  all  the 
judges  in  every  situation.  The  entire  range  of  the 
Scale  has  been  used  by  the  judges  in  every  group,  check 
marks  having  been  found  in  every  one  of  the  five  degrees 
of  the  Scale. 

Results  appear  favorable  in  that  the  judges  evidence 
discrimination  between  traits.  The  writer  does  not, 
nevertheless , venture  to  argue  from  the  data  presented 
in  the  previous  discussion  how  seriously  halo  affects 
the  Vocational  Service  Rating  Scale.  Up  to  the  present 
time  there  is  no  objective  standard  or  numerical  measure 
which  indicates  the  absence  or  presence  of  halo. 

Qualitative  material.—  A fourth  method  of  showing 
validity  of  the  Scale  is  that  of  supporting  observational 
data.  This  material  may  be  obtained  from  the  use  of  the 
Scale  and  from  other  records.  Such  knowledge  may  be 
helpful  in  interpreting  the  results  of  any  one  individual 
rating.  Qualitative  data  is  presented  in  the  following 
Chapter. 
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CHAPTER  VI 


QUALITATIVE  ASPECTS  OF  RATING  SCALES 


Information  from  Use  of  the  Scale 
Need  for  qualitative  approach. — The  statistics  of 
the  Vocational  Service  Rating  Scale  are  presented  in 
Chapter  V,  and  qualitative  material  is  contained  in  this 
Chapter*  Instruments  for  measuring  personality  ordi- 
narily can  not  show  why  a person  made  a particular  score 
but  only  that  he  did  make  the  score. 

1/ 

Kingsley  indicates  reasons  for  understanding  the 
total  person. 

As  valuable  as  are  the  findings  regarding 
the  status  of  a group  derived  from  questionnaires 
or  tests,  and  the  knowledge  of  the  characteristic 
features  of  human  behavior  obtained  from  experi- 
ments, these  are  not  sufficient  for  understanding 
an  individual  child's  particular  difficulties  and 
needs.  Each  child's  personality  presents  a unique 
constellation  of  a large  number  of  traits,  and  the 
product  of  a great  variety  of  hereditary,  environ- 
mental, and  training  influences.  Likewise,  his 
behavior  is  the  result  of  a complex  of  motives  and 
circumstances.  If  a child  is  a behavior  problem, 
or  is  maladjusted  educationally,  socially,  or  emo- 
tionally, adequate  treatment  calls  for  an  under- 
standing of  the  factors  that  have  contributed  to 
his  particular  difficulty.  The  clinical  method 
provides  a diagnosis  of  the  underlying  causes  of 
misbehavior  or  maladjustment.  It  rests  on  the  con- 
viction that  treatment  should  be  concerned  with 
causes  rather  than  symptoms . 

S T H.  L.  Kingsley,  The  Nature  and  Conditions  of  Learning. 
Prentice-Hall.  New  York,  194*),  p.  16. 
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Individual  viewed  In  entirety .--  Raters  should  never 


neglect  the  total  environment  although  they  may  find  it 
necessary  to  give  careful  attention  to  specific  traits 
and  modes  of  behavior.  The  Vocational  Service  Rating 
Scale  does  not  imply  disbelief  in  the  organic  or  Gestalt 
conception  of  personality.  Viewing  the  Individual  as 
a total  organism,  moreover,  may  give  further  meaning 
to  the  traits  listed  on  the  Scale. 

Judges  have  a tendency  to  notice  behavioral  inci- 
dents that  are  of  special  interest  to  them.  With  the 
aid  of  a rating  scale r attention  may  be  focussed  toward 
events  that  are  of  greater  significance  to  the  ratee. 

Many  more  facets  of  personality  than  are  covered  by  the 
traits  on  the  Scale  are  involved  in  getting  to  know 
an  individual.  New  fields  of  personality  exploration, 
however,  may  be  opened  through  the  efforts  of  rating 
traits  helpful  along  vocational  lines. 

Interpretation  of  behavior.--  Raters  became  more 
concerned  over  the  fact  that  incidents  may  be  interpreted 
in  several  different  ways.  They  also  discovered  that 
a single  episode  does  not  necessarily  prove  anything 
about  an  individual.  The  danger  of  lifting  behavior 
experience  out  of  the  social  setting  in  which  it  occurred 
and  reporting  it  in  isolation  was  recognized.  All  of 
the  Judges  became  more  conscious  of  the  background  against 
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which  an  action  took  place  before  allowing  it  to  influ- 
ence their  ratings.  Raters  found  themselves  observing 
an  individual  as  an  organized  whole  and  as  a developing 
personality • 

An  active  search  for  underlying  principles  occurred. 
The  complexities  of  behavior  were  more  clearly  crystalized. 
Hypotheses  were  checked  against  an  organized  framework 
of  explanatory  principles  in  order  to  discover  contra- 
dictory, oversimplified,  or  biased  interpretations. 

Judges  in  school  and  industry  seemed  more  confident 
of  the  accuracy  of  their  ratings  than  those  in  the  nurs- 
ing and  club  situations.  This  fact  appears  reflected 
in  the  statistics  for  these  groups.  Even  when  judges 
do  not  agree  too  closely  on  specific  traits,  useful 
Information  may  be  obtained  about  an  individual  for 
guidance,  placement,  and  personality  improvement. 

When  different  Judges  give  approximately  the  same 
trait  rating  to  a subject,  they  appear  to  have  similar 
information.  When  there  is  disagreement  between  the 
Judges,  there  appears  to  be  a noticeable  difference  in 
the  amount  of  Information  that  each  rater  possesses. 

One  month  after  rating  members  of  the  boy's  club, 
the  three  Judges  were  asked  to  identify  each  ratee  on 
the  basis  of  their  previous  ratings.  The  Judges  unani- 
mously agreed  that  they  looked  for  an  outstandingly 
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high  or  low  trait.  One  judge  correctly  identified  25 
subjects,  another  24,  and  the  third  23* 

Although  judges  mentioned  an  emotional  like  or 
dislike  of  certain  individuals,  their  ratings  did  not 
conform  to  these  opinions.  There  is  a tendency  for 
judges  to  require  a longer  period  of  time  to  rate  traits 
checked  off  toward  the  lower  end  of  the  Scale. 

Use  of  qualitative  material  by  judges. — Discussion 
of  rating  results  brought  forth  a good  deal  of  inter- 
esting material.  The  stimulating  power  of  rating  scales 
for  supposedly  forgotten  incidents  was  a common  expres- 
sion of  the  raters.  They  mentioned  that  various  facts 
brought  into  relief  by  the  Vocational  Service  Rating 
Scale  were  used  to  help  particular  individuals.  Judges 
found  themselves  planning  new  ways  of  helping  people 
under  their  supervision. 

The  club  leaders  became  more  sensitive  to  the  mem- 
bers of  their  group.  Changes  were  evidenced  in  behav- 
ior. One  boy,  who  was  rated  very  low  on  the  trait  so- 
ciability, thought  he  was  acting  sociably.  The  effect 
on  others,  however,  was  quite  different.  Discussion 
of  this  fact  with  the  club  leader  and  the  boy  himself, 
developed  a new  and  positive  status  for  this  youngster 
within  the  club.  The  new  relationship  was  also  carried 
over  to  the  school  situation.  Nearly  all  of  the  boys 
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of  his  club  were  enrolled  in  the  general  course  at  the 
same  high  school. 

In  the  industrial  situation  use  was  made  of  the 
ratings  on  the  Vocational  Service  Rating  Scale.  Certain 
employees  were  called  into  the  personnel  office  for  in- 
terviews. The  supervisors  discussed  with  each  employee 
those  traits  in  which  he  was  rated  relatively  high  and 
then  proceeded  to  those  in  which  he  was  low.  In  some 
instances  discussion  and  suggestions  for  improvement 
brought  about  favorable  changes  in  behavior.  Although 
there  was  no  observable  change  in  overt  behavior  of 
some  of  the  men, there  appeared  to  be  a more  favorable 
working  relationship  between  management  and  the  employees* 

Anecdotal  Material 

Industry Qualitative  aspects  of  rating  scales 

have  not  been  given  sufficient  mention  in  either  school 

or  industry.  In  general, industry  has  employed  this 

type  of  material  to  a greater  extent  than  the  schools* 

General  Foods  Corporation  does  not  use  numerical 

scores  on  their  rating  scale.  The  thinking  behind  their 

rating  policy  is  expressed  by  the  following  quotation 

1/ 

from  the  National  Industrial  Conference  Board* 

1/  Employee  Rating . Studies  in  Personnel  Policy  No. 

39,  National  Industrial  Conference  Board,  New  York, 

1942,  p.  76. 
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It  will  be  noticed  that  no  numerical  values 
appear  on  the  forms.  None  are  used  after  the  forms 
have  been  filled,  nor  are  the  forms  subjected  to 
any  sort  of  statistical  evaluation,  the  management 
believing  that  it  is  neither  desirable  nor  possi- 
ble to  reduce  records  of  judgment  on  human  values 
to  exact  figures.  They  are  recognized  as  being 
approximations  only.  Further,  it  is  not  believed 
that  any  total  numerical  scores  would  be  as  indica- 
tive of  an  individual’s  job  behavior  as  is  the 
picture  revealed  by  the  form  itself,  on  which  each 
personal  characteristic  is  a separate  and  distinct 
item.  1/ 

Although  there  is  much  to  be  said  in  favor  of  this 
point  of  view,  the  use  of  some  scoring  method  facilitates 
further  analysis  and  action.  Most  firms  have  maintain- 
ed and  furthered  the  use  of  scores  even  though  recog- 
nizing that  they  may  be  misleading  if  not  used  with 
care  and  discrimination. 

A large  utility  company  stresses  the  importance  of 
supervisors’  comments  in  substantiating  employees’  rat- 
ings. An  example  from  their  records  is  as  follows. 

"Good  resistance  to  colds  and  no  absence  be- 
cause of  sickness.  The  personal  habits  of  this 
employee  are  very  good. 

"Must  be  closely  supervised  to  see  that  all 
details  have  been  fulfilled. 

"Does  not  cooperate  with  other  employees  when 
it  comes  to  notification  of  certain  changes  rela- 
ting to  their  work.  Must  have  a better  spirit 
of  cooperation  and  a willingness  to  work  with  the 
group  so  that  the  work  can  be  accomplished  more 
effectively ." 

When  this  same  employee  was  called  in  by  his  di- 


1/  Ibid 
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vision  supervisor, 

He  was  complimented  on  (his)  good  showing 
on  knowledge  of  work,  physical  fitness,  and  per- 
sonal habits.  It  was  pointed  out  that  largely 
because  of  his  attitude,  he  was  making  a poor  im- 
pression on  his  supervisors  which  was  not  helping 
hi3  possibilities  for  advancement.  It  was  suggest 
ed  that  (1)  he  apply  himself  fully  to  the  work 
assigned  to  him  and  endeavor  to  keep  it  up  to 
schedule;  (2)  that  when  his  own  work  permits,  he 
assist  the  man  in  the  next  higher  position;  (3) 
that  he  read  current  publications  having  a bear- 
ing on  his  own  or  related  fields  of  work  so  that 
he  will  have  a broader  background  concerning  his 
work.  It  appeared  to  me  that  apparently  these 
suggestions  were  well  received. 

The  example  presented  above  illustrates  practical  anec- 
dotal material.  A possible  course  of  action  that  may 

be  taken  is  also  indicated. 

1/ 

Fuller  stresses  counseling  techniques  in  the  G-ood 
year  Aircraft  rating  program.  Shortcomings  and  unde- 
sirable emotional  traits  may  be  rectified  through  coun- 
sel and  training.  He  points  out  the  value  of  Improving 
personal  characteristics  of  employees  by  discussion  of 
ratings.  Employers  and  employees . also  gain  knowledge 
about  one  another  that  they  did  not  have  prior  to  the 
interview. 


The  boss  told  me  things  I wanted  to  know. 

He  told  me  things  of  which  I did  not  realize  he 
was  aware.  On  the  other  hand  discussion  gave  me 
an  opportunity  to  tell  him  additional  things  which 
I wanted  to  be  sure  he  appreciated.  I wanted  to 
know  how  I was  doing  and  how  I stood  so  that  I 


l/  S.  E.  Fuller,  "Goodyear  Aircraft  Employee  Counseling 
Personnel  Journal:  1944:  23:  145-153;  176-184. 
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would  not  continually  wonder  about  my  performance 
and  future*  I wanted  to  know  so  that  I could  enjoy 
the  satisfaction  that  I was  making  progress*  In 
addition  to  this  I wanted  a plan  of  action  so  that 
I could  correct  my  inabilities  and  in  correcting 
them  get  a better  job*  1/ 

The  current  rating  program  of  the  Washington  Gas 

Light  Company  replaces  a previous  numerical  rating  policy* 

2/ 

Some  of  the  comments  recorded  on  their  rating  forms 
are  as  follows* 

"A  crane  operator’s  foreman  reports,  'The 
quality  of  his  work  has  greatly  improved  lately  •' 

"A  clerical  superior  writes,  ’Is  learning 
to  correct  his  habit  of  excessive  visiting  during 
working  hours • ' 

"A  department  head  comments  regarding  an 
assistant,  ’After  the  need  of  careful  checking 
was  pointed  out  last  year,  his  work  has  definitely 
improved.  Is  also  demonstrating  more  tact  in 
assigning  work  to  his  subordinates •' " 

'2/  ' 

Laney  summarizes  the  company's  experience  with  the 
present  rating  program. 

The  above  statements  taken  from  actual  rat- 
ing forms  typify  our  company's  experience  with 
the  merit  rating  plan  described  in  this  article. 

They  are  part  of  the  mounting  evidence  that  this 
plan  introduced  a year  and  a half  ago  is  getting 
results.  They  are  tangible  proof  of  better  job 
performance,  a primary  objective  of  personnel  ad- 
ministration* 

Schools.--  Schools  in  general  have  realized  the 
1 / Ibid.,  p.  180 • 

2/  A.  R.  Laney,  Jr.  "Getting  Results  from  Merit  Rating", 
Personnel : 1945:  22 j 171* 


1/  Ibid 


value  of  anecdotal  records.  Today  many  schools  provide 
some  space  on  their  rating  scales  where  qualitative 
information  may  be  noted  to  supplement  a numerical  score 
In  the  schools  where  records  are  kept  over  a period  of 
years,  the  various  qualitative  observations  tend  to 
form  a clear  picture.  Many  of  them,  nevertheless,  have 
not  recorded  qualitative  material  to  any  marked  degree. 

A scale  that  mentions  the  value  of  qualitative  ma- 

i / 

terial  is  the  New  York  Rating  Scale  for  School  Habits. 

On  the  back  of  the  rating  sheet  there  are  several  state- 
ments to  be  checked  by  the  teacher.  They  indicate  how 
well  he  knows  the  ratee  and  where  he  has  observed  the 
student's  behavior.  Directions  advise  the  judge  to 
write  items  of  importance  in  explaining  a trait  rating 
on  the  back  of  the  scale. 

The  directions  for  the  American  Council  on  Educa- 

U 

tlon  Personality  Rating  Scale  stress  in  some  detail  the 
use  of  anecdotal  evidence  that  may  support  ratings. 

The  scale  Itself  provides  a liberal  amount  of  space 
to  support  one's  Judgment  in  the  areas  covered  by  the 
scale • 

1/  E.  L.  Cornell,  W.  W.  Coxe,  and  J.  S.  Orleans,  New 
York  Rating  Scale  for  School  Habits.  World  Book  Co. 
Yonkers,  New  York,  1927,  4-  pp. 

2/  American  Council  on  Education  Personality  Rating 
Scale.  American  Council  on  Education,  Washington,  D.  C. 
Revision  B,  1929. 
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A blank  sheet  is  provided  by  the  Teacher*  s Rating 

1/ 

Scales  for  Puoll  Adjustment  for  recording  information 
of  various  types  such  as  special  interests  and  disabili- 
ties as  well  as  anecdotal  records.  More  space  is  fur- 
nished for  qualitative  information  than  in  many  school 
rating  schedules. 

Vocational  Service  Rating  Scale.—  Anecdotes  and 
inf ormation  which  were  derived  from  the  Scale  are  rich 
in  observational  material.  The  information  may  be  ex- 
panded and  used  effectively  to  help  people  not  only 
vocationally  but  also  in  total  personality  improvement. 

A few  anecdotes  obtained  from  using  the  Scale  with  mem- 
bers of  a boy's  club,  high  school  students,  industrial 
workers,  and  student  nurses  are  presented. 

"S.  J.,  one  of  the  finest  boys  at  the  club, 
is  lame.  At  first  he  was  very  conscious  of  it 
and  took  a back  seat.  Cooperation  in  teamwork  and 
superior  fulfillment  of  promises  and  obligations 
won  a place  for  him.  He  now  has  many  friends  and 
promotes  a good  feeling  when  others  are  present." 

"R.  L.  has  outstanding  physical  vitality, 
but  in  the  wrong  direction.  He  is  always  ready 
to  use  his  fists  and  flies  off  the  handle  at  a 
moment's  notice." 

"M.  E.  is  usually  unsuitably  dressed,  and 

her  appearance  is  a slovenly  one.  Her  hair  is 

untidy,  nails  not  filed,  wrinkled  clothes,  and 

generally  untidy.  In  spite  of  these  deficiencies 

she  does  show  brighter  than  average  intelligence 

both  in  school  work  and  in  extra  curricular  acti- 
. . . n 

vities  • 

I T F.  N.  Freeman  and  E.  Kawin,  Teacher's  Rating  Scales 
for  Pupil  Adjustment.  University  of  Chicago  Press. 
Chicago,  1937* 
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11 J.  M.  tends  to  speak  exceptionally  clearly 
and  pleasingly  but  in  so  doing  causes  social  dif- 
ficulties. She  tends  to  monopolize  the  conversa- 
tion, the  result  of  which  brings  about  one  of  the 
reasons  that  she  does  not  make  friends  easily." 

"J.  N.  does  exceedingly  exact  work  at  aLl 
times  and  demands  the  same  of  others;  qualities 
which  make  it  difficult  for  him  in  relation  to 
others  in  the  plant." 

"P.  L.  has  been  told  that  he  will  be  let  go 
if  there  is  not  decided  improvement.  He  can  do 
work  with  above  average  accuracy,  but  he  is  erratic 
and  has  disappointed  the  foreman  on  important  work. 
He  is  over-sensitive  and  does  not  fit  in  well  with 
the  rest  of  the  men." 

"R.  E.  has  many  friends  and  makes  friends 
easily.  Patients  take  a liking  to  her  from  the 
very  beginning." 

"L.  G-. , an  outstanding  girl  and  always  help- 
ful, has  many  friends.  She  shows  superior  self- 
control  on  many  occasions.  Many  doctors  have  com- 
mented upon  this  ability,  and  her  resourcefulness 
under  trying  conditions." 


Summary 

One  need  not  rely  solely  on  a strictly  qualitative 
or  qualitative  approach  in  the  interpretat ion  of  rating 
results.  The  statistical  method  generally  has  received 
the  greater  emphasis.  Qualitative  material,  however, 
may  support  and  help  explain  statistical  data.  Its  pro- 
mise of  usefulness  suggests  inclusion  in  this  study. 
Information  of  this  sort  may  lead  to  further  improvement 
of  the  accuracy  of  Judges'  ratings. 
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CHAPTER  VII 


SUGGESTED  USES  OF  THE  VOCATIONAL  SERVICE  RATING  SCALE 

In  this  Chapter  an  attempt  is  made  to  point  out 
various  ways  in  which  the  Vocational  Service  Rating 
Scale  may  be  used  as  a practical  tool  in  industry  and 
school. 

Use  of  the  Scale  in  Industry 

Objectives . — The  Vocational  Service  Rating  Scale 
may  assist  in  (1)  selecting  new  employees;  (2)  discov- 
ering and  eliminating  unsuitable  employees;  (3)  transfer- 
ing,  promoting,  and  re-employing  personnel;  (4)  help- 
ing foremen  and  supervisors  in  personnel  relations; 
and  (5)  establishing  bases  for  lay-off  and  salary  ad- 
ministration, These  objectives  will  be  discussed  in 
detail  since  the  school  (source  of  employer  and  employee) 
may  also  apply  them  in  guidance,  placement,  and  person- 
ality improvement. 

Selecting  new  employees,--  Trait  scores  of  old 
employees  on  the  Vocational  Service  Rating  Scale  may 
be  one  of  the  bases  for  choosing  new  employees.  Men 
who  select  new  employees  may  become  conscious  of  trait 
ratings  of  previous  successful  workers.  They  may  then 
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be  more  reluctant  to  hire  anyone  who  does  not  approach 
these  personality  standards.  It  is  possible  that  in 
time  significant  job  patterns  may  be  developed. 

Discovering  and  eliminating  unsuitable  employees . — 
Despite  care  in  selecting  new  employees,  some  will  be 
found  unsatisfactory.  In  a few  cases  their  unsuitability 
is  sufficiently  obvious  in  the  early  stages  of  employ- 
ment to  necessitate  immediate  dismissal.  Most  employees, 
however,  should  have  sufficient  merit  to  warrant  a 
thorough  trial.  As  much  as  six  to  nine  months  trial 
may  be  required  before  their  employability  may  be  esti- 
mated accurately. 

It  is  recommended  that  employees  be  rated  frequent- 
ly during  their  first  year  of  service.  Ratings  seem 
advisable  at  the  end  of  the  first,  third,  and  sixth 
month  and  toward  the  end  of  the  first  year.  Three  to 
six  months  experience  and  education  in  a company's  meth- 
ods may  be  required  for  potentially  suitable  employees 
to  become  familiar  with  their  work  environment.  This 
period  of  time  may  also  be  needed  for  supervisors  to 
become  well  acquainted  with  new  employees.  Each  situa- 
tion must  be  carefully  analyzed  in  the  light  of  succes- 
sive ratings  on  the  Scale.  Where  there  is  some  doubt 
concerning  a man's  ultimate  progress,  placing  him  suc- 
cessively under  several  supervisors  may  be  advisable. 
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The  status  of  any  employee  may  be  determined  on 
the  basis  of  his  ratings.  If  he  does  not  meet  the  com- 
pany's standards,  appropriate  measures  may  be  taken 
before  he  becomes  eligible  for  company  benefits.  Un- 
satisfactory employees  may  be  kept  on  the  job  to  the 
detriment  of  the  company  and  the  individual.  Recently 
an  employee  of  20  years  service  was  let  go  by  his  em- 
ployer. The  latter  told  the  employee  that  he  would 
eventually  be  grateful  for  this  action.  With  an  ade- 
quate rating  system  it  is  unlikely  that  such  an  event 
would  have  taken  place . 

Transferlng.  promoting,  and  re-employing  person- 
nel . — Periodic  ratings  on  the  Vocational  Service  Rating 
Scale  may  assist  in  bringing  to  light  employees  who 
are  unable  to  maintain  effectiveness  in  their  customary 
line  of  work  until  they  are  eligible  for  retirement. 
Appropriate  adjustments  to  other  types  of  work  may  be 
made  • 

Employee  ratings  may  indicate  which  men  are  doing 
work  of  sufficiently  high  calibre  to  warrant  promotion. 
Comparable  ratings  between  departments  may  be  establish- 
ed. Employees  may  then  be  selected  for  advancement 
either  departmentally  or  interdepartmentally. 

The  Scale  may  provide  a better  choice  of  men  for 
re-employment.  During  periods  of  extra  work,  vacations, 
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seasonal  fluctuations,  or  where  permanent  increase  in 
personnel  is  contemplated,  previous  ratings  may  furnish 
an  excellent  guide  to  a man's  qualifications. 

Helping  foremen  and  supervisors. — Foremen  and 
supervisors  may  assume  their  full  responsibility  in 
personnel  matters  through  the  use  of  the  Vocational 
Service  Rating  Scale.  In  the  absence  of  an  adequate 
rating  scale,  employees  may  question  the  judgment  and 
fairness  of  their  supervisors.  The  supervisor  may  dis- 
cuss with  an  employee  the  personality  patterns  required 
for  success  on  the  job.  Scores  on  the  Scale  may  make 
it  easier  for  the  supervisors  to  advise  a man  concern- 
ing his  standing  and  how  he  may  improve.  It  is  essen- 
tial that  the  supervisor  be  Intimately  acquainted  with 
his  men.  The  Scale  may  be  the  stimulus  for  acquiring 
the  information  needed. 

Establishing  bases  for  lay-off  and  salary  admin- 
istration . — The  Vocational  Service  Rating  Scale  may 

serve  as  one  of  the  methods  of  establishing  an  equitable 

1/ 

basis  for  lay-off.  Hanawalt  has  pointed  out  the  ad- 
vantages of  merit  rating  for  this  purpose.  Reliable 
ratings  with  company-wide  comparability  may  facilitate 
selection  for  lay-off  of  the  least  qualified  men. 

1 / W.  R.  Hanawalt,  "Solving  the  Problem  of  Merit  vs. 
Seniority  in  Lay-Off",  Personnel : 194-7*  23:  405-4-09 • 
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The  Scale  furnishes  records  of  personality  patterns 
that  may  assist  in  effecting  equitable  salary  administra- 
tion. This  Information  may  be  particularly  helpful  in 
the  administration  of  unclassified  employees. 

Cooperation  of  participants.—  Before  the  Scale  may 
be  used  successfully,  raters  and  ratees  must  be  in  favor 
of  the  program.  No  rating  program  will  succeed  unless 
it  maintains  the  active  cooperation  and  understanding 
of  the  participants.  Some  concerns  have  unsuccessful 
rating  programs  because  all  details  of  administering 
and  interpreting  the  schedules  are  placed  on  a "top 
secret"  basis.  Others  which  let  employees  know  the 
objectives  of  the  rating  program  are  more  successful. 

Conducting  the  rating  interview. — It  is  the  re- 
sponsibility of  management  to  select  the  proper  indi- 
vidual to  conduct  rating  interviews.  Each  situation 
will  differ  as  no  two  individuals  are  alike.  The  method 
of  handling  the  details  may  be  designed  to  fit  the  par- 
ticular circumstances. 

In  interpreting  the  results  on  the  Scale,  the  su- 
pervisor should  select  a time  and  place  where  both  will 
be  mentally  and  physically  at  ease.  The  supervisor's 
criticism  should  be  constructive.  He  should  show  in- 
terest in  the  employee's  development  and  success.  This 
role  is  not  an  easy  one.  The  supervisor  may  have  to 
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cope  with  varying  degrees  of  receptivity  from  genuine 
appreciation  to  outright  resentment.  The  supervisor, 
at  the  same  time,  must  try  to  stimulate  the  employee 
to  improve  his  ratings.  The  employee  must  realize  that 
his  personality  assets  will  be  brought  to  the  attention 
of  management. 

Discussion  of  ratings. — An  employee  may  assume 
that  he  is  getting  along  satisfactorily  if  no  one  has 
advised  him  to  the  contrary.  Some  employees  are  par- 
tially satisfied  with  this  Indirect  assurance.  Other 
employees  in  the  early  stage  of  their  employment  feel 
quite  insecure.  They  either  do  not  know  with  whom  to 
discuss  their  problems  or  are  hesitant  to  approach  their 
supervisors.  Still  others  wish  some  definite  informa- 
tion on  their  progress  and  have  no  hesitation  in  seek- 
ing it. 

Discussion  of  ratings  with  men  of  relatively  short 
service  should  usually  be  quite  detailed  and  comprehen- 
sive. Adequate  guidance  at  this  time  may  have  the  great- 
est effect  on  their  progress.  This  does  not  mean  that 
long  service  men  should  not  receive  the  same  benefits. 
Many  long  term  workers  who  have  become  problems  may 
attain  satisfactory  ratings  through  constructive  help. 

If  time  does  not  permit  discussion  of  ratings  with 
all  employees,  discussion  should  not  be  limited  to  those 
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with  low  ratings.  High  ranking  men  may  improve  still 
further  or  at  least  have  the  satisfaction  of  knowing 
they  are  doing  a commendable  job.  There  is  positive 
value  in  discussing  a man's  outstanding  qualities  before 
going  on  to  details  of  low  trait  scores.  It  helps  the 
employee  to  maintain  his  self-respect  and  more  favor- 
ably approach  his  lower  ratings. 

Training  of  raters. — In  previous  Chapters  it  was 
recommended  that  a minimum  of  three  judges  be  used. 

In  large  concerns  it  may  prove  advisable  to  appoint 
one  rating  counselor  for  each  department.  He  will  have 
general  supervision  of  rating  procedures  and  training 
of  supervisory  personnel.  Ratings  can  be  no  better 
than  the  understanding  and  interest  of  the  raters  them- 
selves. It  is  the  counselor's  duty  to  educate  raters 
by  means  of  the  Vocational  Service  Rating  Scale  Manual. 
Actual  practice  and  subsequent  review  and  analysis  of 
their  work  is  the  method  suggested. 

1/ 

The  following  quotation  of  Hopkins  can  not  be 
too  strongly  emphasized. 

We  are  forced  to  accept  what  is  so  pointed- 
ly implied  — that  if  a merit  rating  plan  is  no 
better  than  the  raters  who  use  it,  then  in  order 
to  have  a really  effective  plan,  the  supervisory 
staff  must  undergo  an  intensive  training  course 

1/  J.  T.  Hopkins,  "Some  Fallacies  and  Virtues  in  Merit 
Rating" , American  Management  Association  Production 
Series ; 1940:  124:  30-39. 
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in  the  appraisal  of  men. 

It  is  extremely  important  that  the  counselor  be 
a person  who  is  respected  by  his  department  personnel. 
He  should  be  familiar  with  his  duties  as  well  as  the 
objectives  and  principles  of  rating. 

Rating  committees.—  One  of  three  types  of  rating 
committee  will  fit  the  needs  of  most  industrial  con- 
cerns. In  small  companies  it  may  be  necessary  for 
ratings  to  be  made  by  single  individuals.  This  is 
the  least  desirable  and  least  reliable  method.  This 
plan  eliminates  an  opportunity  for  discussion  which 
helps  to  educate  raters.  It  also  offers  little  com- 
pensation for  personal  bias  or  poor  judgment. 

The  second  type  of  committee  is  composed  of  a 
neutral  chairman  and  at  least  three  supervisors  of 
different  rank.  The  supervisors  must  be  personally 
acquainted  with  the  work  of  the  ratees.  Unless  care- 
fully guarded  against,  the  opinions  of  the  higher 
ranking  supervisors  will  have  undue  influence  on 
ratings  and  disposition.  In  this  form  of  committee, 
written  or  oral  discussion  should  start  with  the 
lowest  ranking  supervisors.  This  committee  is  pre- 

17  Ibid. 
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ferable  to  the  one  man  committee 


The  third  type  of  committee  is  considered  the  best 
procedure  and  is  usually  the  most  satisfactory  to  raters 
and  rat6es.  It  is  composed  of  a neutral  chairman  and 
at  least  three  supervisors  of  equal  rank.  The  super- 
visors must  be  personally  acquainted  with  the  work  of 
the  ratees.  This  type  of  committee  follows  the  rules 
of  rating  and  promotes  frank  and  open  discussion.  It 
prevents  deference  to  the  opinions  of  a Judge  of  higher 
rank  and  is  easier  for  uniform  training  of  raters.  This 
is  considered  the  best  type  of  rating  committee  and  should 
be  used  whenever  possible. 

Recording  ratings. — Methods  of  recording  indus- 
trial ratings  vary  from  one  concern  to  another.  They 
depend  to  a great  extent  on  company  procedures  and  poli- 
cies for  record  keeping.  Recording  of  ratings  on  the 
Scale  may  well  fit  into  the  existing  procedures  for 
record  keeping. 

Use  of  the  Scale  in  Education 
Alms .--  It  is  hoped  that  the  Vocational  Service 
Rating  Scale  may  be  used  in  schools  for  guidance,  place- 
ment, and  personality  improvement.  Recommended  uses 
of  the  Scale  mentioned  earlier  in  this  Chapter  have 
application  in  school  systems.  Principles  of  rating 
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procedures  apply  equally  as  well  to  use  of  the  Scale 
In  education.  Since  schools  may  prepare  youth  well 
in  advance  of  their  becoming  employees  or  employers, 
the  value  of  such  training  is  significant. 

1/ 

Guidance  and  placement.—  Jones  has  stressed  the 
school1 s importance  in  assisting  the  student  in  the  area 
of  vocational  placement.  When  employers  request  per- 
sonality evaluation  of  a student,  it  is  extremely  im- 
portant that  such  information  be  available. 

2/ 

Hughes  expresses  employers’  interest  in  the  per- 
sonality of  prospective  employees. 

"When  the  business  man  wishes  to  employ  a 
person,  he  does  not  as  a general  rule  care  a great 
deal  about  how  much  learning  the  person  possesses. 

He  wants  to  know  whether  the  youth  is  easily  dis- 
couraged, whether  he  is  subject  to  prejudices, 
whether  he  is  a loafer  or  a doer.  He  wants  to 
know  whether  the  lad  avoids  or  accepts  responsi- 
bilities, whether  he  behaves  enthusiastic  over 
his  work  or  lacks  energy,  whether  he  mistrusts 
his  own  Judgment  or  relies  upon  it. 

"Needless  to  say,  if  these  human  qualities 
are  important  for  success,  schools  must  plan  Just 
as  definitely  for  developing  them  as  for  giving 
information." 

The  need  for  maintaining  permanent  personality  re- 
cords has  often  been  expressed  by  teachers.  In  some  schools 

1/  A.  J.  Jones,  Principles  of  Guidance.  McGraw  Hill. 

New  York,  1934,  second  edition,  p.  355. 

2/  W.  H.  Hughes,  "General  Principles  and  Results  of 
Rating  Trait  Characteristics" , Journal  of  Educational 
Method:  1925:  4:  430-431. 
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there  is  nothing  on  the  permanent  record  card  that  in- 
dicates the  personality  of  the  individual.  It  is  then 
difficult  to  find  someone  who  knew  the  former  student 
well  enough  to  note  and  remember  his  personality.  Per- 
sonality evaluation  based  upon  ratings  on  the  Vocation- 
al Service  Rating  Scale  may  fulfill  both  the  need  ex- 
pressed by  employers  and  teachers.  The  data  contained 
in  Tables  7 and  8 may  be  part  of  the  student's  perma- 
nent record. 

Personality  improvement.—  Research  has  shown  that 
personality  characteristics  may  be  changed.  The  modern 
way  of  viewing  personality  is  known  as  the  social-skill 
concept.  Personality  is  described  in  terms  of  a per- 
son's ability  to  get  along  well  with  people.  This  view 
of  personality  does  not  rest  upon  the  inheritance  of 
either  attractive  or  weak  personal  traits.  It  suggests 
that  personality  is  placed  upon  an  attainable  basis. 

The  possibility  of  personality  improvement  is  a strong 
incentive  for  young  people. 

One  of  the  possible  techniques  for  helping  students 

improve  personality  is  through  motivation  of  learning. 

1/ 

Kingsley  points  out  five  general  ways  for  motivating 

learning.  They  are  (1)  creating  the  desire  for  the 

1 / H.  L.  Kingsley,  The  Nature  and  Conditions  of  Learn- 
ing. Prentice-Hall,  New  York,  1946,  pp.  92-103. 
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outcomes  of  learning;  (2)  enlistment  of  motives  already 
present;  (3)  engendering  favorable  attitudes;  (4)  fos- 
tering the  development  of  ideals;  and  (5)  establishing 
tasks  and  problems.  They  are  not  mutually  exclusive 
and  as  far  as  possible  should  be  used  together.  These 
principles  may  be  applied  to  the  field  of  personality. 

Creating  the  desire  for  the  outcomes  of  learning.— 
A desire  for  the  outcomes  of  learning  should  be  created. 
Significance  is  given  to  the  task,  and  it  is  an  effective 
means  for  obtaining  diligent  effort.  The  results  that 
are  to  be  eventually  attained  must  be  valued. 

The  teacher  may  stress  the  relation  of  personality 
traits  in  the  classroom  to  the  business  world.  The 
importance  of  personality  traits  in  the  world  of  work 
may  be  easily  and  convincingly  shown  through  unit  ac- 
tivites.  Teachers  may  mention  concrete  examples  where 
trait  characteristics  and  subject  matter  dovetail. 

The  Vocational  Service  Rating  Scale  may  be  used 
with  boys  and  girls  beginning  with  the  ninth  grade.  At 
this  time  youth  is  starting  to  view  the  world  of  work 
in  a more  serious  vein.  If  rating  is  properly  present- 
ed to  students  as  an  aid  in  their  future  careers  and 
not  as  a "blacklist,"  enthusiasm  may  be  gained  for  the 
program.  This  has  been  shown  in  schools  at  Rockland 
County  and  Dobbs  Ferry  and  in  others. 
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Enlistment  of  motives  already  present. — The  enlist- 
ment of  motives  already  present  is  another  method  for 
motivating  learning.  It  is  important  that  the  teacher 
harness  the  basic  needs  of  the  student  to  his  school 
activities . 

Artificial  incentives,  praise  and  reproof,  know- 
ledge of  results,  and  the  like  may  be  used,  especially 
when  benefits  or  values  may  be  attained  only  after  a 
long  period  of  time.  One  may  place  more  faith  in  praise 
than  in  blame.  Both  may  be  used  as  motivation  for  de- 
sired conduct  rather  than  as  a means  of  meting  out  jus- 
tice. Very  little  faith  may  be  placed  in  artificial 
rewards  such  as  prizes,  gold  stars,  and  fancy  badges. 

Most  students  are  fundamentally  interested  in  their 
own  welfare.  Favorable  rating  of  traits  on  the  Scale 
may  appeal  to  motives  that  are  already  present.  Most 
high  school  students  want  to  be  a part  of  the  group 
and  to  make  and  to  hold  friends.  Traits  such  as  socia- 
bility, cooperation,  reliability,  appearance,  and  the 
like  may  take  on  a new  meaning  when  viewed  in  the  light 
of  their  needs.  The  classroom  situation  may  be  rela- 
ted to  a larger  one  in  the  world  of  work.  The  student, 
for  example,  who  enjoys  and  is  highly  accurate  in  all 
of  his  work  may  be  shown  the  importance  of  this  trait 
in  various  types  of  employment. 
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Engendering  favorable  attitudes.--  A third  method 


of  motivating  learning  is  by  engendering  favorable  atti- 
tudes. The  teacher  should  endeavor  to  prevent  or  remove 
such  negative  attitudes  as  fear  of  failure  and  feelings 
of  inferiority.  He  should  actively  try  to  build  up 
feelings  of  security  and  a healthy  self-confidence. 

The  teacher  should  not  attempt  more  than  the  student 
is  capable  of  achieving.  Students  should  know  what 
is  expected  of  them.  They  should  be  informed  of  their 
progress  not  just  once  or  twice  a semester  but  whenever 
the  occasion  arises. 

One  of  the  best  methods  of  trying  to  improve  an 
individual's  weak  traits  on  the  Vocational  Service  Rat- 
ing Scale  is  to  initially  point  out  where  he  shows 
strength.  When  patterns  of  strength  have  been  recog- 
nized by  the  counselee,  the  counselor  may  then  proceed 
to  those  traits  that  need  correction  or  improvement. 

It  is  best  to  begin  working  with  a few  students  who 
want  and  need  the  teacher's  help.  It  is  also  a wise 
procedure  to  modify  only  a few  traits  at  a time.  Every 
effort  should  be  made  to  see  that  the  student  achieves 
a sample  of  success  relatively  soon. 

Fostering  the  development  of  ideals.—  A fourth 
method  of  motivating  learning  is  fostering  the  develop- 
ment of  ideals.  Ideals  may  be  expected  to  promote  learn- 
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ins?  since  they  are  generalized  controls  of  activity* 

If  the  student  is  made  to  realize  the  importance  of 
ideals  in  his  work,  he  may  try  to  develop  them  in  order 
to  live  and  work  more  efficiently*  The  ideal  of  re- 
liability , for  example,  may  be  carried  from  the  class- 
room to  another  situation  where  reliability  is  recog- 
nized as  an  ideal  rather  than  as  a habit* 

Establishing  tasks  and  problems. — The  fifth  method 
of  motivating  learning  is  that  of  establishing  tasks 
and  problems.  Exercises  assigned  by  the  teacher  should 
be  such  that  the  activities  they  induce,  may  result 
in  improvement  of  traits  on  the  Vocational  Service  Rat- 
ing Scale.  To  be  effective,  these  exercises  must  be 
understandable  at  the  ratee's  level. 

Recording  ratings. — Various  methods  of  recording 
pupils’  personality  may  be  employed.  The  general  mechani- 
cal framework  of  a plan  is  described  in  some  detail. 

It  is  suggested  that  pupils  be  rated  at  least  twice 
during  a semester  by  preferbly  three  different  judges. 

The  first  rating  should  take  place  by  the  close  of  the 
first  nine  weeks  of  school.  The  final  rating  should 
occur  before  final  grades  are  recorded.  At  the  begin- 
ning of  the  semester  each  judge  should  have  a Scale 
sheet  for  every  student.  Tentative  ratings  may  be  re- 
corded on  the  Scale  throughout  the  year.  At  the  end 
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of  the  term  summarization  of  these  ratings  may  be  a 
simple  matter.  Time  saved  and  accuracy  maintained  on 
the  permanent  final  ratings  may  more  than  Justify  fre- 
quent notations. 

The  rater  may  retain  a copy  of  the  Vocational 
Service  Rating  Scale  for  each  student  as  a work  sheet. 
Whenever  a student  exhibits  behavior  pertinent  to  the 
traits  on  the  Scale,  the  Judge  may  check  the  proper 
point  on  the  Scale.  The  sequence  of  letters  indicates 
the  order  in  which  the  check  marks  are  recorded.  This 
procedure  may  provide  a record  of  growth  or  decline 
in  the  particular  trait.  At  the  close  of  the  semes- 
ter, one  of  the  trait  scales  may  look  like  Figure  1. 

Figure  1 
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If  the  teacher  has  a large  number  of  students,  it 
may  be  difficult  or  inconvenient  to  maintain  a Scale  at 
his  desk  for  each  student.  In  this  case,  the  counselor 
may  keep  one  copy  as  a work  sheet  for  several  students. 
Whenever  a student  exhibits  behavior  pertinent  to  the 
traits  on  the  Scale,  the  teacher  may  record  a check  mark 
with  the  initials  of  the  student  for  whom  the  rating  is 
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The  first  rating  on  the  Scale  is  indicated  by  placing 
the  letter  "a"  after  the  student* s initials.  On  the 
next  rating  the  letter  Hb"  is  used,  and  successive  let- 
ters indicate  the  order  in  which  the  entries  are  record- 
ed. When  the  work  sheet  appears  crowded,  the  teacher 
may  transfer  the  scores  to  the  Scale  for  each  student. 

The  same  procedure  is  followed  for  each  successive  work 
sheet.  Scales  marked  in  the  manner  described  above  may 
look  as  pictured  in  Figure  2. 


Figure  2 
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The  judge *s  average  score  on  each  trait  for  each 
student  may  be  calculated  at  the  end  of  the  semester. 
Ratings  may  be  summarized  either  by  averaging  numerical 
values  or  by  choosing  the  point  on  the  Scale  that  ap- 
pears to  represent  a summary  of  them.  Each  student* s 
average  for  each  Judge  may  be  compared  with  that  of 
the  group.  The  recommended  record  sheet  for  one  Judge's 
rating  is  shown  in  Table  7.  The  form  for  combined  scores 
of  all  Judges  is  shown  in  Table  8. 
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Table  7 


School  Hating  Record 

Date: Rated  by: 

Semester: Title  of  Rater: 

Individual  Score  Group  Score 

Appearance  

Voice  and  Speech  

Ability  to  Present  Ideas  

Physical  Vitality  

Intelligence  

Alertness  

Accuracy  

Leadership  . 

Initiative  

Emotional  Stability  

J udgment  

Cooperation  

Sociability  

Reliability  


Self-confidence 
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Table  8 

School  Rating  Record 


Date : 

Semester: 


Judges:  (A) 

Name 

IB) 

1 CJ 


Title 


Appearance 


Judge  A 

Judge  B 

Judge  C 

Indivi- 

Indivi- 

Indivi 

dual  Group 

dual  Group 

dual 

Group 

Score  Score 

Score  Score 

Score 

Score 

Voice  and 
Speech 

Ability  to  Pre- 
sent Ideas 

Physical 

Vitality 

Intelligence 

Alertness 


Accuracy 

Leadership  

Initiative  


Emotional 

Stability 

Judgment 


Cooperation 

Sociability 

Reliability 

Self-confidence 


■!  ' ' ' S <'  ' • 


: r.‘  ” 


tAl  :•  ■ lv:r. 


i ; 


: Jy\ 


; 


. • • - • ' 

ir/  - I c r--'  ' • ' 

/21.V  - .I  S'SS 


30X:j  ‘laeCQi. 

b! .a  eoioV 
Jo  C-X  >■ 

- "S'  r y : 1 ; \. 
e it  : 1'  >t  x n 

Ifivt-tV 

lodrl 
' j \tf:  ’I A 

2 ; ' . : ) - 
' 

1 3.:cI.to:.sL 
•\  : f * 

Jr-  ’■ fl~  'Jt> 
• r - ■ 
'■( 

- 


Summary 


This  discussion  concerning  the  use  of  the  Vocation- 
al Service  Rating  Scale  in  industry  and  school  serves 
merely  as  a general  outline.  Those  using  the  Scale 
may  adopt  practices  that  will  fit  their  personnel  needs. 
The  ingenuity  of  those  who  use  the  Scale  will  determine 
its  ultimate  value. 
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CHAPTER  VIII 


SUMMARY  AND  CONCLUSIONS 
Scope  of  the  Study 

Purpose The  objective  of  this  study  was  the  con- 
struction of  a reliable  and  valid  rating  scale  with  sug- 
gested uses  for  vocational  guidance,  placement,  and  person- 
ality improvement*  Measures  were  extended  to  attain  the 
demonstrated  advantages  of  previous  research  and  at  the 
same  time  eliminate  or  modify  many  of  their  technical 
weaknesses.  Special  emphasis  was  placed  on  the  adequate 
training  of  raters  through  the  use  of  a concise  and  com- 
prehensive manual* 

Limitations  * — In  every  research  study  there  are 
a number  of  considerations  which  possibly  limits  its 
value.  Although  investigators  are  aware  of  the  limita- 
tions of  the  research,  they  continue  their  preliminary 
efforts • 

In  spite  of  the  care  with  which  the  Vocational 
Service  Rating  Scale  Manual  was  constructed,  refinements 
may  be  included  in  further  studies.  Since  the  Manual 
is  the  primary  source  of  instruction  for  raters,  pos- 
sible improvements  in  its  total  efficiency  should  be 
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considered 


Further  concrete  and  detailed  examples  explaining 
the  rules  of  rating  may  be  included.  Raters  should  be 
able  to  see  relevant  and  irrelevant  elements  in  these 
illustrations.  It  is  suggested  that  rating  rules  in- 
clude the  provision  that  judges  receive  practice  rating 
on  a trial  group.  During  this  time,  errors  should  be 
corrected  as  they  occur.  This  may  prevent  a tendency 
for  raters  to  read  and  understand  instructions  yet  fail 
to  follow  them. 

Trait  explanations  may  be  so  reworded  that  they 
have  further  working  efficiency.  Although  it  is  unlike- 
ly that  any  definition  will  mean  the  same  thing  to  all 
people,  certain  improvements  may  be  fostered.  Addition- 
al refinements  of  trait  terminology  may  be  introduced. 
Trait  illustrations  may  be  so  constituted  that  Judges 
note  the  differences  between  relevant  and  irrelevant 
details.  In  so  far  as  possible  traits  should  be  defined 
in  non-technical  language  that  is  clearly  understandable 
at  the  raters'  level. 

The  Manual  is  the  primary  tool  for  the  Instruction 
of  raters.  Refinements  of  the  Manual  should  be  accepted 
only  after  it  has  been  demonstrated  that  the  Judges  are 
able  to  do  a better  Job  of  rating. 

Questions  raised. — This  investigation  was  subjected 
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to  analysis  in  order  to  ascertain  solutions  to  the  fol- 
lowing problems* 

(1)  How  reliable  is  the  Vocational  Service  Rating 
Scale? 

(2)  How  valid  is  the  Vocational  Service  Rating 
Scale? 

(3)  Is  the  reliability  and  validity  of  the  Voca- 
tional Service  Rating  Scale  considerably  increased  when 
the  number  of  Judges  is  doubled? 

(4)  How  Important  a role  do  rating  scales  play 

in  personnel  relations  with  special  reference  to  industry? 

(5)  What  major  weakness  may  be  discovered  from  an 
analysis  of  numerous  rating  scales  with  particular  refer- 
ence to  industry? 

(6)  How  well  do  judges  in  different  situations 
agree  in  the  rating  of  the  same  traits? 

(7)  How  may  the  Vocational  Service  Rating  Scale 
be  employed  in  a program  of  guidance , placement , and 
personality  improvement? 

Conclusions 

The  conclusions  will  be  discussed  in  the  same  order 
as  the  problems  were  brought  up  for  attention. 

(1)  The  reliability  coefficients  for  the  Judges 
in  the  boy's  club  situation  ranged  from  a correlation 
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of  .62  with  a probable  error  of  .083  on  trait  15  to  a 
correlation  of  .83  with  a probable  error  of  .042  on 
trait  14. 

The  reliability  coefficients  for  the  judges  in  the 
high  school  group  ranged  from  a correlation  of  .85  with 
a probable  error  of  .038  on  traits  1 and  3 to  a corre- 
lation of  .96  and  a probable  error  of  .011  on  trait  10. 

The  reliability  coefficients  for  the  judges  in  the 
Industrial  situation  ranged  from  .73  with  a probable 
error  of  .063  on  traits  6 and  15  to  a correlation  of 
.89  with  a probable  error  of  .028  on  traits  4 and  8. 

The  reliability  coefficients  in  the  nursing  group 
ranged  from  .66  on  traits  3,  10,  11,  12,  and  15  with  a 
probable  error  of  .07 6 to  .88  with  a probable  error 

of  .031  on  trait  14. 

1/ 

Traxler  mentions  that  reliability  coefficients 
for  most  personality  tests  are  between  .70  and  .90. 
There  are  reliability  coefficients  as  low  as  .54  in  his 
list  of  32  more  useful  personality  tests  and  inventor- 
ies • 

(2)  This  writer  leans  toward  the  thinking  of  Rem- 

mers,  McCall,  Bradshaw,  and  others,  who  believe  that 

validity  may  be  shown  by  comparing  the  ratings  of  dif- 

I/  a.  e:  Traxler,  Techniques  of  Guidance.  Harper  & 
Bros.  New  York,  1945,  p.  107* 
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ferent  Judges  on  the  same  individuals.  Acceptance  of 
this  concept  warrants  the  statement  that  favorable  va- 
lidity is  evidenced  by  the  Vocational  Service  Rating 
Scale,  Validity  coefficients  are  sufficiently  high  in 
all  four  situations. 

Nevertheless,  in  a field  as  controversial  as  that 
of  rating,  three  other  methods  were  also  used  to  demon- 
strate the  validity  of  the  Vocational  Service  Rating 
Scale,  They  are  the  index  of  reliability,  trait  dis- 
crimination by  the  Judges  in  spite  of  possible  halo 
effect,  and  supporting  qualitative  data. 

The  index  of  reliability  is  in  reality  a validity 
coefficient.  It  tells  how  well  a test  is  measuring 
true  ability  in  the  function  which  it  purports  to  measure. 
The  coefficients  ranged  from  .79  on  trait  15  to  .91  on 
traits  2,  8,  and  14  in  the  boy's  club  group;  from  .92 
on  traits  1 and  3 to  .98  on  trait  10  for  the  high  school 
situation;  from  .85  on  traits  6 and  15  to  .94  on  traits 
4 and  8 for  Industry;  and  from  .81  on  traits  3,  10,  11, 

12,  and  15  to  .94  on  trait  14  for  the  nurses.  Of  the 
60  possible  coefficients,  30  fall  at  a correlation  of 
•90  or  better,  and  29  range  from  .80  to  .89,  The  index 
of  reliability  in  all  four  situations  is  sufficiently 
high  to  warrant  the  statement  that  favorable  validity 
is  evidenced  by  the  Vocational  Service  Rating  Scale. 
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The  range  of  scores  for  those  subjects  who  scored 
highest  or  lowest  on  one  trait  were  compared  with  each 
other  and  with  the  remainder  of  the  group.  Trait  dis- 
crimination was  evidenced  in  all  three  groups.  The 
greatest  discrimin&ion  was  often  exercised  in  the  high- 
low  groups  where  halo  by  definition  should  have  been 
the  greatest.  While  the  halo  effect  may  operate  to  in- 
fluence the  scores  on  individual  traits,  it  is  not  so 
great  as  to  prevent  the  judges  from  exercising  consid- 
erable discrimination. 

Qualitative  data  accompanying  the  Vocational  Ser- 
vice Rating  Scale  seems  to  corroborate  ratings  of  the 
judges.  The  material  is  discussed  in  detail  in  Chap- 
ter VI. 

(3)  When  the  number  of  judges  is  doubled  for  pre- 
diction of  reliability  coefficients  by  the  Spearman- 
Brown  Prophesy  Formula,  the  results  are  as  follows. 

In  the  boy's  club  correlations  ranged  from  .76  on 
trait  15  to  .91  on  trait  14;  for  the  high  school  stu- 
dents from  .92  on  traits  1 and  3 to  .98  on  trait  10; 
for  the  Industrial  group  from  .84  on  traits  6 and  15 
to  .94  on  traits  4 and  8;  and  for  the  nurses  from  .80 
on  traits  3,  10,  11,  12,  and  15  to  .93  on  trait  14. 
Fifty-nine  correlations  are  over  .80,  of  which  30  are 
.90  or  over.  When  the  number  of  Judges  is  doubled. 
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the  reliability  and  validity  of  the  Scale  is  considerably 
increased. 

(4)  Rating  scales  play  an  important  role  in  person- 
nel work.  This  is  especially  true  in  industry.  Of  253 
companies  contacted,  57  per  cent  replied  that  rating 
scales  are  a part  of  their  personnel  procedures. 

(5)  A study  of  numerous  rating  scales  reveal  that 
the  most  common  major  defect  is  the  lack  of  an  adequate 
manual  of  instructions.  In  a detailed  study  of  35  in- 
dustrial rating  scales,  only  five  are  accompanied  by  a 
manual  of  instructions. 

(6)  Traits  that  show  relatively  high  agreement  be- 
tween judges  in  one  situation  tend  to  show  similar  agree- 
ment in  others.  The  sae  trend  is  evidenced  when  the 
agreement  of  judges  is  not  as  high.  Differences  are  con- 
siderably lessened  when  the  number  of  judges  is  increased 
There  are  certain  exceptions  and  modifications. 

Trait  15  appears  to  be  the  most  difficult  to  rate 
in  all  groups  except  in  the  high  school.  Trait  14  shows 
the  highest  reliability  coefficient  for  judges  in  the 
boy's  club  and  nursing  situations.  The  high  school  judge 
find  their  greatest  agreement  on  trait  10.  The  highest 
agreement  of  the  Judges  in  industry  occurred  on  traits  4 
and  8.  The  Judges  in  the  nursing  group  find  it  most  dlf- 
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flcult  to  agree,  with  the  boy’s  club,  industry,  and 
high  school  following  in  that  order. 

(7)  There  are  different  ways  of  using  the  Vocation- 
al Service  Rating  Scale  in  a program  of  guidance,  place- 
ment, and  personality  improvement.  An  adequate  method 
in  one  situation  will  not  necessarily  prove  satisfactory 
in  another.  The  plan  developed  should  be  in  line  with 
one's  needs.  Suggested  general  and  specific  techniques 
both  in  regard  to  the  mechanical  and  psychological 
approach  are  described  in  Chapter  VIII. 

Suggestions  for  Further  Study 

The  present  study  has  suggested  further  explora- 
tion in  the  field  of  rating  scales. 

(1)  Are  the  reliability  and  validity  of  judges' 
ratings  significantly  increased  when  they  receive  in- 
tensive training  in  the  use  of  the  Vocational  Service 
Rating  Scale? 

(2)  What  is  the  coefficient  of  correlation  when 
Judges  in  different  situations  rate  the  same  subjects? 

(3)  Are  there  significant  differences  in  the  rat- 
ings of  judges  depending  upon  the  sex  of  the  rater  and 
ratee? 

(4)  How  do  the  ratings  students  receive  as  fresh- 
men compare  with  those  they  receive  as  seniors? 
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(5)  How  closely  do  teachers'  ratings  of  senior  high 
school  students  compare  with  those  of  employers  on  the 
students'  first  jobs? 

(6)  Are  there  significant  differences  between  two 
equated  groups,  one  trained  and  the  other  untrained 

in  methods  of  trait  improvement  on  the  Vocational  Ser- 
vice Rating  Scale? 
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APPENDIX 


VOCATIONAL  SERVICE  RATING  SCALE 


Name  Date  Year 

Last  First  Middle 

Person  Rating 

Instructions:  Keeping  the  definition  of  the  trait  in  mind,  rate  the  individual  between 
minimum  and  maximum  by  placing  a check  mark  (y  ) on  the  proper  scale  at  the  point  where 
in  your  judgment,  the  individual  belongs.  Base  your  estimates  solely  on  evidence.  Try 
not  to  omit  ratings  of  any  traits. 


MINIMUM  RATING 

MAXIMUM  RATING 

APPEARANCE 

i . i 

i 

i 

l 

1 

i 

0 Unsuitable 

I 1 

1 Creates  rather 
unfavorable 
impression 

1 - i * H 

2 Suitable  or 
acceptable 

i 

3 Creates  distinct- 
ly favorable 
impress  ion 

l 

• 4 Impress ive,  com- 
mands admiration 

5 

apN 

VOICE  AND  SPEECH 

0 Irritating  or 
indistinct 

i 

| Understandable 
but  rather  un- 
pleasant 

i i 

2 Neither  conspicu- 
ously pleasant 
nor  unpleasant 

i i 

3 Def  initely 
pleasant  and 
distinct 

l i 

If  Exceptionally 
clear  and 
pleasing 

i 

5 

m 

PHYSICAL 

VITALITY 

0 Little  physical 
endurance 

1 i 

| Below  average 
in  physical 
vitality 

i 

2 Suitable  or 
acceptable 

i i 

3 Above  average 
H 1 i . 

If  Outstanding 
physical 
vitality 

i 

5 

1 

ALERTNESS 

0 Slow  in  grasping 
the  obvious 

1 

|Slow  in  grasping 
meaning  of 
interviewer' s 
questions 

i - i 

2 Nearly  always 
grasps  intent  of 
interviewer' s 
questions 

i i 

3 Rather  quick  in 
grasping  ques- 
tions and  new 
facts 

1 MM 

If  Exceptionally 
keen  and  under- 
standing 

i m a 

5 

i v! 

ACCURACY 

0 Does  inexact 
work 

1 J 

| Below  average 
in  accuracy 

J i 

2 Suitable  or 
acceptable 

j i 

3 Work  accuracy 
above  average 

IH  , 

If  Accomplishes  very 
exact  work 

i 

5 

■ 

INITIATIVE 

0 Lacks  intellec  - 
tual  curiosity. 
Seldom  starts 
anything  new 

| Be  low  average  in 
finding  means  to 
overcome  diffi- 
culties 

2 Suitable  or 
acceptable 

3 Above  average 
in  overcoming 
difficulties 

if  Starts  new 
undertakings 

5 

ABILITY  TO 

1 r. 

I i . 

i i 

i i 

i.  i - 

. 1 

PRESENT  IDEAS 
RELIABILITY 

0 Confused  and 
illogical 

1 

| Tends  to  become 
involved 

i 

2 Usually  gets  his 
ideas  across 
well 

i i 

3 Shows' superior 
ability  to  ex- 
press himself 

H J H 1 

If  Unusually 
logical 

i 1 , 1 

5 

'I:?. 

0 Neglects  pro- 
mises and 
obligations 

1 1 

| Of  ten  late  at 
appointments 

1 1 

2 Generally  ful- 
fills promises 
and  obligations 

i i 

3 Above  average 
in  fulfilling 
obligations  and 
promises 

i i 

if  Very  dependable 
in  word  and  deed 

1: 

5 

COOPERATION 

0 Avoids  group  ac- 
tivities. Seems 
unhappy  in  groups 

1 

| Usually  does  not 
subordinate  self 
to  group 

i i 

2 Takes  part  in 
normal  number  of 
group  activities 

i i 

3 Participates 
frequently  in 
teams  and  groups 

i 

If  Seems  happy  in 
team  work.  Subor- 
dinates self 
to  group 

i aH 

5 

9 ■ 

SELF-CONFIDENCE 

0 Very  timid  and 
hesitant 

1  i 

| Little  confidence 
in  himself 

i 

2 Moderately  con- 
fident of  himself 

i 

3 Wholesomely 
self-conf  ident 

i 

If  Excellent  self- 
assurance 

i 

5 

1 

LEADERSHIP 

0 Prefers  follow- 
ing plans  of 
others 

1  i 

| Of  ten  fails  to 
secure  support 
of  his  ideas 

l 

2 Occasionally 
takes  leading 
part  in  group 
activities 

i 

3 Often  takes 
leading  part 
in  group 
activities 

i 

if  Plans  for  and 
directs  others 
successfully 

5 

1 

EMOTIONAL 

STABILITY 

0 Over  sensitive 
feelings  easily 
hurt 

1 r 

| Occasionally 
flies  off  the 
handle 

2 Well  poised  most 
of  the  time 

i 

3 Superior  self 
command 

1 1 

if  Exceptional 
poise,  calmness 
and  good  humor 
under  stress 

i 

5 

J 

SOCIABILITY 

0 Keeps  people  at 
a distance 

1 I 

|Does  not  make 
friends  easily 

i 

2 Likeable 

1 i 

3 Has  many  friends 
1 

4Has  a great  many 
close  friends 

i 

5 

1 

JUDGMENT 

0 Lacks  balance 
and  restraint 

i 

|Some  tendency 
to  react  with- 
out restraint 

1 i 

2 Acts  judiciously 
in  ordinary  cir- 
cumstances 

1 

3 Has  habit  of 
considered 
judgment 

1 

4 Inspires  unusual 
confidence  in 
soundness  of 
judgment 

i 

5 

1 

INTELLIGENCE 

0 Very  dull 

|Not  very  bright 

2 Shows  average 
intelligence 

3 Brighter  than 
average 

if  Exceptionally 
intelligent 

5 

Table  9 

BoyTs  Club 
Judge  A 

Raw  Scores  on  Vocational  Service  Rating  Scale 


Traits 

Sub- 


.iects 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

1 

19 

12 

10 

14 

22 

21 

24 

21 

25 

12 

16 

42 

29 

29 

20 

2 

33 

37 

27 

34 

20 

21 

22 

15 

10 

25 

25 

31 

35 

26 

22 

3 

31 

27 

28 

39 

29 

34 

39 

28 

30 

25 

19 

40 

35 

32 

40 

4 

25 

22 

21 

22 

30 

30 

36 

22 

22 

22 

22 

37 

37 

40 

28 

5 

24 

15 

25 

27 

35 

35 

35 

29 

28 

18 

25 

30 

29 

29 

30 

6 

26 

22 

28 

27 

39 

38 

31 

34 

25 

36 

14 

26 

37 

23 

35 

7 

25 

27 

22 

28 

32 

32 

37 

15 

15 

18 

24 

40 

31 

40 

20 

8 

27 

27 

20 

36 

30 

25 

25 

19 

19 

20 

20 

25 

39 

30 

38 

9 

44 

44 

38 

35 

40 

30 

26 

40 

39 

42 

36 

43 

43 

38 

38 

10 

43 

43 

40 

39 

42 

38 

38 

30 

26 

25 

24 

27 

36 

28 

35 

11 

28 

27 

14 

27 

19 

19 

19 

7 

7 

15 

15 

15 

25 

15 

25 

12 

38 

30 

38 

35 

37 

35 

44 

35 

25 

34 

35 

44 

44 

45 

35 

13 

26 

39 

44 

24 

44 

45 

44 

45 

44 

36 

36 

46 

34 

44 

44 

14 

35 

35 

24 

36 

34 

34 

40 

25 

24 

24 

21 

37 

41 

42 

35 

15 

39 

37 

36 

25 

37 

37 

25 

26 

25 

38 

38 

42 

44 

44 

35 

16 

26 

26 

15 

39 

25 

21 

16 

16 

16 

16 

4 

22 

36 

5 

38 

17 

11 

19 

16 

16 

6 

6 

12 

4 

11 

19 

5 

6 

14 

5 

17 

18 

12 

13 

12 

17 

16 

16 

14 

10 

10 

18 

14 

16 

14 

11 

8 

19 

26 

26 

26 

38 

38 

38 

30 

49 

36 

25 

16 

26 

26 

26 

46 

20 

25 

31 

11 

25 

25 

25 

15 

15 

6 

14 

15 

25 

25 

15 

5 

21 

35 

36 

35 

25 

36 

26 

35 

34 

35 

40 

41 

41 

41 

41 

24 

22 

26 

26 

34 

26 

25 

13 

37 

37 

36 

20 

30 

40 

39 

39 

25 

23 

38 

38 

37 

37 

25 

30 

16 

17 

17 

15 

16 

25 

34 

25 

24 

24 

38 

35 

38 

38 

37 

44 

37 

36 

36 

15 

37 

37 

45 

45 

26 

25 

18 

24 

24 

41 

25 

25 

25 

36 

31 

30 

26 

49 

35 

36 

36 
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Table  10 

Boy's  Club 
Judge  B 

Raw  Scores  on  Vocational  Service  Rating  Scale 


Traits 

Sub- 


.iects  1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

1 

21 

15 

4 

30 

25 

30 

28 

29 

36 

15 

20 

49 

30 

39 

26 

2 

26 

26 

20 

32 

20 

16 

25 

10 

6 

36 

26 

26 

32 

20 

26 

3 

35 

31 

22 

35 

26 

36 

31 

10 

25 

26 

17 

26 

40 

36 

40 

4 

25 

6 

26 

26 

35 

36 

45 

25 

26 

36 

30 

36 

30 

49 

30 

5 

26 

25 

26 

31 

26 

35 

26 

22 

25 

25 

23 

36 

36 

30 

36 

6 

7 

6 

26 

32 

20 

36 

25 

50 

26 

35 

15 

11 

23 

26 

50 

7 

26 

26 

27 

26. 

36 

39 

46 

10 

30 

30 

41 

50 

36 

50 

21 

8 

30 

26 

15 

40 

25 

36 

24 

10 

25 

20 

20 

28 

40 

26 

40 

9 

46 

43 

36 

40 

43 

36 

30 

41 

36 

41 

36 

48 

50 

50 

41 

10 

41 

46 

43 

35 

36 

36 

39 

32 

30 

26 

31 

46 

40 

36 

30 

11 

26 

23 

18 

24 

23 

26 

20 

10 

26 

20 

20 

16 

36 

22 

26 

12 

26 

17 

20 

30 

36 

40 

43 

26 

26 

36 

41 

46 

46 

50 

36 

13 

36 

50 

47 

27 

50 

50 

39 

36 

50 

43 

48 

50 

40 

45 

46 

14 

26 

35 

28 

40 

31 

30 

39 

28 

30 

26 

28 

40 

36 

30 

36 

15 

36 

31 

26 

26 

30 

30 

35 

22 

26 

26 

30 

40 

41 

40 

48 

16 

13 

16 

13 

40 

20 

16 

20 

15 

26 

11 

15 

20 

26 

18 

40 

17 

19 

25 

20 

16 

11 

8 

11 

6 

3 

21 

8 

20 

25 

18 

18 

18 

18 

21 

10 

25 

15 

19 

11 

6 

1 

30 

11 

6 

11 

6 

8 

19 

26 

28 

30 

41 

31 

30 

36 

45 

30 

30 

15 

20 

26 

18 

50 

20 

21 

32 

20 

25 

25 

21 

11 

6 

1 

30 

10 

6 

30 

1 

40 

21 

30 

36 

24 

25 

36 

27 

36 

40 

36 

46 

50 

50 

40 

48 

21 

22 

22 

25 

20 

37 

26 

20 

40 

32 

36 

41 

31 

50 

36 

50 

20 

23 

46 

50 

31 

35 

30 

31 

26 

25 

16 

16 

26 

35 

45 

30 

50 

24 

45 

36 

40 
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48 
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50 

40 

20 
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50 

50 

50 
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20 

30 
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25 

21 
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41 
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49 

40 

47 
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Table  11 

Boy's  Club 
Judge  C 

Haw  Scores  on  Vocational  Service  Rating  Scale 


Traits 

Sub- 

.iects  1 2 3 4 5 6 7 S 9 10  11  12  13  14.  15 


1 

25 

13 

13 

32 

25 

20 

25 

19 

17 

17 

23 

34 

28 

24 

24 

2 

34 

33 

27 

35 

31 

30 

27 

24 

16 

25 

23 

31 

37 

23 

22 

3 

23 

25 

40 

44 

30 

36 

40 

30 

28 

11 

13 

34 

35 

38 

38 

4 

34 

21 

21 

26 

28 

28 

27 

16 

22 

20 

26 

32 

27 

41 

22 

5 

33 

27 

27 

28 

38 

37 

31 

29 

22 

18 

22 

34 

34 

29 

22 

6 

22 

24 

40 

25 

33 

40 

30 

40 

32 

35 

4 

22 

23 

22 

33 

7 

26 

29 

33 

28 

34 

38 

33 

23 

28 

18 

21 

44 

12 

43 

12 

8 

33 

30 

30 

34 

31 

25 

25 

22 

19 

19 

23 

31 

35 

24 

28 

9 

45 

45 

48 

34 

39 

35 

31 

38 

31 

40 

31 

40 

44 

41 

29 

10 

45 

42 

43 

33 

37 

37 

34 

27 

31 

25 

34 

35 

38 

42 

19 

11 

29 

23 

20 

26 

20 

18 

6 

1 

1 

25 

7 

1 

13 

12 

9 

12 

41 

30 

41 

33 

46 

49 

40 

38 

31 

25 

34 

42 

50 

44 

42 

13 

29 

38 

49 

27 

39 

50 

43 

43 

36 

35 

32 

44 

50 

44 

39 

14 

40 

40 

47 

36 

35 

50 

41 

35 

42 

19 

10 

46 

42 

36 

14 

15 

39 

33 

35 

22 

36 

37 

34 

23 

26 

47 

47 

44 

47 

41 

20 

16 

32 

32 

35 

43 

35 

34 

25 

25 

36 

13 

7 

23 

37 

27 

34 

17 

21 

26 

18 

24 

23 

10 

16 

16 

25 

20 

22 

19 

22 

27 

22 

18 

16 

21 

13 

20 

21 

19 

18 

11 

12 

21 

11 

21 

14 

20 

13 

19 

21 

29 

22 

39 

25 

25 

21 

49 

31 

25 

21 

27 

29 

21 

44 

20 

23 

30 

20 

22 

25 

19 

21 

19 

24 

19 

14 

21 

25 

17 

16 

21 

34 

40 

45 

25 

31 

29 

37 

36 

41 

40 

39 

44 

37 

44 

21 

22 

26 

30 

43 

38 

35 

37 

44 

37 

42 

33 

44 

40 

44 

35 

32 

23 

30 

43 

40 

35 

24 

36 

28 

17 

31 

22 

32 

26 

38 

22 

27 

24 

39 

35 

43 

35 

33 

44 

36 

32 

37 

7 

39 

33 

43 

42 

16 

25 

28 

24 

28 

40 

24 

29 

28 

30 

30 

32 

28 

37 

39 

42 

23 
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Table  12 

High  School  Students 
Judge  A 

Raw  Scores  on  Vocational  Service  Rating  Scale 


Traits 


Sub' 

.iects 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

1 

21 

21 

25 

30 

31 

31 

30 
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20 

15 

15 

30 

35 
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25 
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35 

40 

34 

50 

45 
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35 
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Table  13 


High  School  Students 
Judge  B 

Raw  Scores  on  Vocational  Service  Rating  Scale 


Traits 

Sub- 


.iects 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 
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20 

21 

25 

29 
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32 

30 

22 

20 
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Table  14 

High  School  Students 
Judge  C 

Raw  Scores  on  Vocational  Service  Rating  Scale 


Traits 


Sub- 

jects 

1 

2 

3 

4 

5 

6 
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8 
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Table  15 

Industry- 
Judge  A 

Haw  Scores  on  Vocational  Service  Rating  Scale 
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Table  16 


Industry 
Judge  B 

Raw  Scores  on  Vocational  Service  Rating  Scale 


Traits 

Sub- 


.iects 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

1 

37 

35 

45 

42 

45 

45 

35 

45 

50 

15 

30 

40 

40 

25 

47 

2 

28 

25 

21 

25 

25 

35 

37 

18 

35 

35 

20 

25 

15 

25 

30 

3 

25 

25 

30 

30 

35 

35 

30 

30 

35 

40 

35 

25 

30 

25 

35 

4 

25 

20 

32 

35 

25 

25 

30 

25 

35 

10 

30 

25 

25 

25 

30 

5 

38 

40 

40 

40 

30 

30 

30 

30 

35 

35 

25 

40 

43 

35 

30 

6 

21 

25 

45 

35 

38 

25 

30 

35 

15 

25 

10 

30 

30 

15 

15 

7 

24 

25 

25 

25 

30 

30 

35 

20 

35 

30 

35 

35 

35 

35 

30 

8 

26 

27 

20 

35 

25 

23 

35 

15 

32 

20 

25 

32 

35 

25 

20 

9 

25 

25 

25 

30 

25 

30 

35 

30 

35 

25 

30 

30 

30 

25 

25 

10 

22 

15 

18 

25 

25 
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45 
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33 
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Table  17 

Industry 
Judge  C 

Raw  Scores  on  Vocational  Service  Rating  Scale 


Traits 

Sub- 

.iects  1 2 3 4 5 6 7 8 9 10  11  12  13  14  15 

1 40  40  40  40  43  40  35  45  45  IB  35  45  35  25  40 

2 35  31  23  25  27  29  35  10  25  25  25  25  20  25  28 

3 30  30  30  25  35  30  23  25  30  35  30  25  35  22  30 

4 30  28  30  35  30  30  25  35  35  6 25  19  30  20  28 

5 40  25  35  37  33  25  28  25  27  42  29  46  30  30  32 

6 25  29  18  38  40  18  28  30  10  23  12  25  25  15  20 

7 30  29  28  20  34  31  35  25  25  33  35  30  32  42  35 

8 35  35  15  31  33  20  38  10  27  21  30  28  35  30  30 

9 32  29  22  28  27  25  30  25  29  18  25  25  25  30  24 

10  29  22  15  20  26  25  42  20  19  29  45  35  27  35  15 

11  25  18  15  15  29  20  33  22  30  15  30  18  12  23  30 

12  26  29  20  23  30  32  35  10  30  27  25  30  30  30  23 

13  27  32  16  30  15  15  9 10  8 11  6 17  20  7 12 

14  25  30  35  33  35  35  30  35  25  15  28  20  15  20  35 

15  36  33  30  27  40  34  28  30  30  18  32  35  18  30  26 

16  30  18  17  25  29  30  20  10  25  30  16  25  20  23  30 

17  35  35  25  26  35  35  35  20  30  32  28  35  43  30  35 

18  34  35  25  32  30  39  27  22  22  25  35  27  35  20  34 

19  23  42  25  25  38  35  31  40  32  25  29  34  30  33  33 

20  27  32  15  35  25  22  25  35  20  30  24  25  33  25  28 

21  30  30  25  20  25  25  26  10  25  21  25  35  35  30  25 

22  28  30  25  25  43  30  37  41  35  20  34  30  18  27  28 

23  28  15  17  25  25  17  15  35  25  20  15  17  25  15  37 

24  33  19  22  30  30  25  24  15  25  17  30  25  25  25  25 

25  32  33  26  40  25  35  35  25  25  25  27  32  30  34  35 
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Table  18 

Nurses 
Judge  A 

Haw  Scores  on  Vocational  Service  Rating  Scale 


Traits 

Sub- 


.iects 

1 

2 

3 

4 . 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

1 

5 

21 

5 

20 

15 

13 

11 

9 

11 

13 

18 

21 

15 

11 

19 

2 

11 

5 

11 

25 

25 

23 

25 

18 

25 

17 

20 

21 

11 

19 

19 

3 

25 

35 

35 

32 

35 

34 

37 

28 

30 

33 

35 

40 

35 

26 

39 
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21 

35 

19 

24 

27 

29 

25 

23 
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21 

20 

29 

17 

22 
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24 

45 

45 

34 

45 
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40 

30 
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31 
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45 
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31 

13 

19 
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18 
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Table  19 

Nurses 
Judge  B 

Raw  Scores  on  Vocational  Service  Rating  Scale 


Traits 

Sub- 


.iects 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

1 

8 

19 

15 

19 

15 

12 

13 

19 

15 

20 

9 

24 

20 

19 

9 

2 

19 

15 

25 

24 

17 

25 

20 

30 

21 

27 

21 

18 

15 

21 

24 

3 

30 

31 

40 

33 

38 

42 

35 

42 

35 

40 

25 

35 

31 

29 
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Table  20 

Nurses 
Judge  C 

Raw  Scores  on  Vocational  Service  Rating  Scale 


Traits 


Sub- 


.iects 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

1 

11 

15 

9 

14 

18 

11 

17 

20 

11 

24 

20 

25 

25 

12 

2 

2 

15 

10 

20 

25 

12 

20 

23 

24 

29 

20 

21 

15 

18 

19 

29 
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32 

35 

29 

40 

35 

40 

31 

40 

33 

41 
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30 
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31 
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21 
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40 

30 

35 

31 

29 

32 

40 

39 

32 

9 

19 

11 

20 

25 

19 

21 

25 

33 

25 

19 

26 

30 

19 

35 

18 

10 

30 

14 

25 

34 

41 

40 

29 

39 

35 

19 

25 

39 

35 

35 

35 

11 

22 

16 

28 

22 

35 

29 

35 

21 

25 

21 

40 

31 

29 

33 

29 

12 

20 

11 

28 

25 

31 

20 

8 

25 

18 

30 

31 

25 

21 

10 

27 

13 

13 

10 

11 

21 

21 

18 

15 

21 

19 

10 

11 

25 

15 

11 

11 

14 

25 

34 

35 

30 

28 

31 

25 

34 

30 

26 

31 

30 

31 

35 

29 

15 

9 

15 

25 

21 

21 

21 

25 

15 

20 

25 

20 

25 

15 

24 

19 

16 

21 

25 

20 

29 

25 

20 

25 

25 

20 

19 

15 

25 

33 

10 

25 

17 

35 

22 

30 

25 

30 

25 

35 

25 

20 

25 

25 

30 

25 

34 

25 

18 

24 

28 

20 

25 

19 

20 

10 

21 

20 

21 

11 

20 

25 

14 

21 

19 

25 

22 

35 

30 

35 

35 

45 

35 

40 

35 

35 

35 

30 

45 

30 

20 

10 

12 

15 

20 

25 

20 

5 

15 

10 

20 

10 

10 

20 

5 

20 

21 

20 

15 

20 

20 

25 

20 

20 

25 

20 

25 

15 

35 

35 

25 

23 

22 

20 

25 

34 

35 

32 

35 

30 

30 

35 

30 

25 

25 

10 

30 

31 

23 

24 

25 

35 

31 

35 

25 

25 

35 

25 

35 

25 

25 

35 

35 

40 

24 

10 

8 

5 

25 

25 

15 

15 

25 

20 

25 

15 

15 

12 

15 

25 

25 

20 

20 

20 

15 

18 

15 

5 

20 

20 

12 

15 

15 

25 

11 

25 
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Table  21 
Arithmetic  Mean 

Each  Trait  for  Each  Judge  in  Each  Situation 

Judges 

fralts  ABC  A B C 


Boy^  Club High  School  Students 


1 

28.72 

27.44 

30.72 

27.08 

27.52 

26.88 

2 

28.72 

27.80 

30.52 

28.68 

28.16 

27.04 

3 

26.52 

24.88 

32.84 

25.44 

26.12 

25.56 

4 

30.00 

32.40 

31.36 

34.04 

34.32 

31.04 

5 

29.92 

28.84 

31.12 

30.80 

31.28 

29.12 

6 

28.72 

30.12 

32.48 

29.44 

29.96 

28.88 

7 

28.88 

30.08 

29.64 

27.40 

27.56 

26.40 

8 

25.80 

24.96 

27.20 

26.04 

26.68 

26.44 

9 

23.92 

26.08 

27.64 

25.00 

25.00 

24.52 

10 

24.08 

29.08 

24.44 

24.60 

25.04 

25.52 

11 

22.96 

26.84 

24.28 

25.00 

25.20 

25.40 

12 

32.48 

• 34.16 

32.20 

29.32 

28.80 

29.80 

13 

33.92 

35.40 

33.72 

31.48 

31.00 

29.64 

14 

30.12 

33.40 

32.04 

26.36 

26.44 

25.04 

15 

29.16 

33 .76 

24.84 

24.84 

25.68 

25.00 

Industry 

Nurses 

1 

27.68 

25.12 

30.60 

21.40 

22.56 

21.20 

2 

25.84 

24.56 

29.16 

33.56 

22.92 

20.48 

3 

35.96 

26.64 

23 .76 

23.04 

24.88 

23.64 

4 

30.80 

29.68 

28.40 

24.20 

25.60 

26.08 

5 

31.32 

28.64 

31.28 

26.52 

26.88 

27.08 

6 

29.04 

28.00 

28.08 

25.40 

25.52 

24.08 

7 

30.96 

31.80 

29.16 

23.20 

22.64 

22.80 

8 

22.16 

26.56 

24.40 

23.24 

27.00 

26.88 

9 

29.00 

31.16 

26.36 

23.40 

24.96 

23.72 

10 

20.92 

23.72 

23.24 

23.96 

24.96 

24.88 

11 

28.52 

24.20 

27.00 

21.16 

22.52 

22.16 

12 

26.92 

28.48 

28.32 

25.04 

26.12 

25.88 

13 

24.36 

27.60 

27.32 

25.12 

26.12 

25.96 

14 

24.88 

24.32 

25.84 

21.36 

24.36 

24.08 

15 

27.28 

26.84 

28.72 

26.72 

27.12 

26.04 

Table  22 

Standard  Deviation 

Each  Trait  for  Each  Judge  in  Each  Situation 


Judges 


Traits 

A 

B 

C 

A 

B 

C 

Boy*s  Club 

Hiph  School  Students 

1 

8.51 

10.05 

7.67 

10.38 

10.18 

10.70 

2 

8.63 

11.49 

7.65 

8.70 

9.22 

9.38 

3 

9.89 

9.69 

11.08 

8.14 

8.43 

8.61 

4 

7.80 

7.45 

6.61 

8.98 

8.72 

9.38 

5 

8.93 

8.69 

6.41 

9.99 

10.13 

9.37 

6 

9.43 

9.86 

10.23 

9.73 

9.63 

10.21 

7 

9.69 

10.23 

8.92 

9.50 

9.39 

8.94 

8 

11.63 

13.96 

10.80 

10.64 

10.46 

10.96 

9 

10.36 

12.18 

9.52 

9.49 

10.00 

10.66 

10 

8.78 

9.28 

8 .60 

11.04 

10.82 

10.89 

11 

9.89 

11.69 

11.61 

10.68 

10.24 

10.45 

12 

10.70 

14.79 

10.32 

9.75 

9.63 

10.45 

13 

8.16 

8.77 

10.75 

8.99 

8.85 

9.18 

14 

12.11 

14.43 

10.03 

11.22 

10.75 

10.77 

15 

10.14 

11.01 

9.39 

9.24 

10.29 

10.29 

Industry 

Nurses 

1 

6.50 

5.02 

4.4B 

9.55 

9.71 

6 .89 

2 

9.57 

7.07 

6.51 

9.47 

7.19 

8.51 

3 

10.08 

8.72 

6 .86 

10.41 

7.82 

7.99 

4 

9.48 

6.75 

6.52 

5.59 

4.55 

6.37 

5 

6.87 

6.89 

6.42 

9.21 

9.04 

7.93 

6 

6.97 

5.35 

6.55 

8.33 

9.69 

8.24 

7 

7.50 

6.36 

7.31 

8.98 

11.33 

10.68 

8 

9.41 

9.25 

10.66 

9.80 

9.47 

7.57 

9 

7.94 

8.13 

7.31 

7.73 

7.33 

8.41 

10 

10.57 

9.15 

7.88 

7.12 

7.31 

7.29 

11 

7.34 

7.57 

8.00 

7.43 

6.89 

8.25 

12 

8.79 

6.24 

7.54 

8.15 

7.71 

6.95 

13 

10.61 

8.59 

7.41 

9.45 

8.53 

8.31 

14 

9.30 

6.94 

7.24 

8.22 

11.05 

11.90 

15 

6.43 

6.21 

6.49 

8.33 

8.61 

8.52 

Table  23 


Boy's  Club 
Judge  A 

Highest  and  Lowest  Trait  Scores  of  Each  Subject 


Sub- 

jects 

High 

Trait 

Number 

Low 

Trait 

Number 

Spread 

1 

42 

12 

10 

3 

32 

2 

37 

2 

10 

9 

27 

3 

40 

12,  15 

19 

11 

21 

4 

40 

14 

21 

3 

19 

5 

35 

5,  6,  7 

15 

2 

20 

6 

39 

5 

14 

11 

25 

7 

40 

12,  14 

15 

8,  9 

25 

8 

39 

13 

19 

8,  9 

20 

9 

44 

1,  2 

26 

7 

18 

10 

43 

1.  2 

24 

11 

19 

11 

28 

1 

7 

8,  9 

21 

12 

45 

14 

25 

9 

20 

13 

46 

12 

24 

4 

22 

14 

42 

14 

21 

11 

21 

13 

44 

13,  14 

25 

4,  7,  9 

19 

16 

39 

4 

4 

11 

35 

17 

19 

2,  10 

4 

8 

15 

18 

18 

10 

8 

15 

10 

19 

49 

8 

16 

11 

33 

20 

31 

2 

5 

15 

26 

21 

41 

11,  12, 
13,  14 

24 

15 

17 

22 

40 

12 

13 

6 

27 

23 

38 

1,  2 

15 

10 

23 

24 

45 

13,  14 

15 

10 

30 

25 

49 

12 

18 

1 

31 
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Table 

24 

Boy' s 

Club 

Judge 

B 

Highest 

and 

Lowest  Trait 

Scores 

of  Each  Subject 

Sub- 

Trait 

Trait 

jects 

High 

Number 

Low 

Number 

Spread 

1 

49 

12 

4 

3 

45 

2 

36 

10 

6 

9 

30 

3 

40 

13,  15 

10 

8 

30 

4 

49 

14 

6 

2 

43 

5 

36 

12,  13 

22 

8 

14 

15 

6 

50 

8,  15 

6 

2 

44 

7 

50 

12,  14 

10 

8 

40 

8 

40 

4,  13,  15 

10 

8 

30 

9 

50 

13,  14 

30 

7 

20 

10 

46 

2,  12 

26 

10 

20 

11 

36 

13 

10 

8 

26 

i 

12 

50 

14 

17 

2 

33 

13 

50 

2,  5,  6 

27 

4 

23 

9,  12 

14 

40 

4,  12 

26 

1,  10 

14 

15 

48 

15 

22 

8 

26 

16 

40 

4,  15 

11 

10 

29 

17 

25 

2,  13 

3 

9 

22 

18 

30 

10 

1 

9 

29 

19 

50 

15 

15 

11 

35 

20 

40 

15 

1 

9,  14 

39 

21 

50 

11,  12 

21 

15 

29 

22 

50 

12,  14 

20 

3,  6,  15 

30 

23 

50 

2,  15 

16 

9,  10 

34 

24 

50 

8,  12 

20 

10 

30 

13,  14 

25 

49 

12 

13 
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Table  25 
Boy's  Club 
Judge  C 

Highest  and  Lowest  Trait  Scores  of  Each  Subject 


Sub- 

jects 

High 

Trait 

Number 

Low 

Trait 

Number 

Spread 

1 

34 

12 

13 

2,  3 

21 

2 

37 

13 

16 

9 

21 

3 

44 

4 

11 

10 

33 

4 

41 

14 

16 

8 

25 

5 

38 

5 

18 

10 

20 

6 

40 

3,  6,  8 

4 

11 

36 

7 

44 

12 

12 

13,  15 

32 

8 

35 

13 

19 

9,  10 

16 

9 

4£ 

3 

29 

15 

19 

10 

45 

1 

19 

15 

26 

11 

29 

1 

1 

8,  9,  12 

28 

12 

50 

13 

25 

10 

25 

13 

50 

6,  13 

27 

4 

23 

14 

50 

6 

10 

11 

40 

15 

47 

10,  11 
13 

20 

15 

27 

16 

43 

4 

7 

11 

36 

17 

27 

14 

10 

6 

17 

18 

21 

2,  5, 
10,  12 

11 

8,  11 

10 

19 

49 

8 

21 

I,  7, 

II,  14 

28 

20 

30 

2 

14 

11 

16 

21 

45 

3 

21 

15 

24 

22 

44 

7,  11,  13 

26 

1 

18 

23 

43 

2 

17 

8 

26 

24 

44 

6 

7 

10 

37 

25 

42 

14 

23 

15 
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Table  26 


High  School  Students 
Judge  A 

Highest  and  Lowest  Trait  Scores  of  Each  Subject 


Sub- 

jects 

High 

Trait 

Number 

Low 

Trait 

Number 

Spread 

1 

35 

13 

15 

10,  11 

20 

2 

50 

4 

34 

3 

16 

3 

40 

1,  2,  4 

5 

10 

35 

4 

40 

4-6,  14 

25 

3,  10 

15 

5 

20 

2 

5 

10-12,  14 

15 

6 

40 

4,  5 

20 

3 

20 

7 

40 

4,  9-12 
15 

30 

3,  5,  6 
8 

10 

8 

35 

1 

15 

8,  9,  11 
15 

20 

9 

40 

1,  2 

20 

8,  9 

20 

10 

30 

1,  2 

15 

8-11,  15 

15 

11 

50 

8 

40 

1-3,  11 
15 

10 

12 

35 

5-7 

5 

1 

30 

13 

30 

4,  12,  13 

10 

3,  8-11 
14 

20 

14 

40 

4 

10 

12,  14 

30 

15 

40 

4-8,10-13 

25 

1-3 

15 

16 

30 

1,  2,  4-6 
13 

20 

3,  7-12 
14,  15 

10 

17 

50 

4-6,  8 
13,  14 

40 

15 

10 

18 

40 

4-6,  12 
13 

20 

7 

20 

19 

40 

4 

30 

5-7,  9-11 
14,  15 

10 

20 

35 

4 

15 

1-3,  5-9 
11,  14,  15 

20 

21 

40 

4 

30 

10-12 

10 

22 

30 

1-3,  13 

20 

4-12,  14 
15 

10 

23 

30 

4,  12-14 

15 

1 

15 

24 

25 

4,  5 

15 

10,  11 

10 

25 

35 

5-7 

10 

1 

25 

Table  27 


High  School  Students 
Judge  B 

Highest  and  Lowest  Trait  Scores  of  Each  Subject 


Sub- 

jects 

High 

Trait 

Number 

Low 

Trait 

Number 

Spread 

1 

35 

13 

16 

10,  11 

19 

2 

50 

4 

35 

3,  9,  12 

15 

3 

40 

1,  4 

5 

10 

35 

4 

42 

6 

25 

10 

17 

5 

20 

2 

5 

11,  12,  14 

15 

6 

42 

5 

23 

1-3,  15 

19 

7 

42 

4 

30 

3,  5 

12 

8 

23 

1 

10 

8,  9,  15 

13 

9 

39 

2 

22 

8,  9 

17 

10 

32 

1,  2 

15 

10 

17 

11 

50 

8 

40 

11 

10 

12 

33 

5,  6 

3 

1 

30 

13 

32 

4,  13 

10 

9,  10 

22 

14 

40 

4 

11 

14 

29 

15 

44 

8 

25 

1 

19 

16 

35 

1 

18 

9-11 

17 

17 

50 

5,  13 

43 

3 

7 

18 

42 

6 

22 

7 

20 

19 

40 

4 

28 

14 

12 

20 

35 

4 

13 

1-3,  8,  9 

22 

21 

40 

3,  4 

28 

11 

12 

22 

30 

13 

18 

15 

12 

23 

32 

13,  14 

18 

1,  9 

14 

24 

35 

15 

20 

10,  11 

15 

25 

30 

4-7 

17 

3 
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Table  28 


# 


High  School  Students 
Judge  G 

Highest  and  Lowest  Trait  Scores  of  Each  Subject 


Sub- 

jects 

High 

Trait 

Number 

Low 

Trait 

Number 

Spread 

1 

31 

6 

15 

10 

16 

2 

40 

4 

30 

1,  14,  15 

10 

3 

40 

1 

5 

10 

35 

4 

40 

5,  6,  8 
12 

25 

2 

15 

5 

20 

13 

5 

4,  9-12,  14 

15 

6 

40 

5 

20 

1,  2,  9 

20 

7 

45 

15 

27 

7 

18 

8 

38 

1 

10 

8,  9 

28 

9 

45 

2 

25 

10-12 

20 

10 

25 

1-3,  12 
13 

13 

9 

12 

11 

49 

14 

38 

1,  2,  3 

11 

12 

32 

11 

8 

1 

24 

13 

31 

13 

10 

8-11,  14 

21 

14 

40 

4 

15 

12,  14 

25 

15 

45 

13 

20 

3 

25 

16 

36 

6 

20 

7,  9,  13-15 

16 

17 

50 

1-3,  6 
8,  12-14 

41 

4,  5 

9 

18 

41 

4,  12 

21 

8,  9 

20 

19 

40 

4 

30 

1,  2,  5-7 
10,  11,  13- 

10 

15 

20 

31 

12,  13 

10 

1,  2 

21 

21 

40 

4 

30 

2,  3,  12 

10 

22 

31 

1,  2 

20 

4,  7,  11 
12 

11 

23 

35 

12 

11 

1 

24 

24 

26 

2,  3,  4 

16 

9,  10 

10 

25 

36 

4,  11-13 

20 

3 

16 
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Table  29 

Industry 
Judge  A 

Highest  and  Lowest  Trait  Scores  of  Each  Subject 


Sub- 

jects 

Hleh 

Tr*alt 

Number 

Low 

Trait 

Number 

Spread 

1 

48 

9 

11 

10 

37 

2 

40 

7 

11 

8,  15 

29 

3 

44 

10 

22 

8 

22 

4 

40 

4 

3 

10 

37 

5 

48 

2,  13 

24 

8 

24 

6 

46 

4 

17 

15 

29 

7 

38 

11 

14 

8 

24 

8 

38 

13 

11 

8 

27 

9 

37 

4 

18 

15 

19 

10 

46 

7 

15 

3 

31 

11 

31 

5,  7 

9 

10,  13 

22 

12 

42 

14 

12 

8 

30 

13 

39 

4 

6 

12 

33 

14 

39 

4 

9 

10 

30 

15 

41 

6 

7 

13 

34 

16 

35 

4 

9 

3 

26 

17  • 

41 

13 

23 

8 

18 

18 

34 

2,  4 

14 

14 

20 

19 

44 

5 

20 

1 

24 

20 

37 

4 

21 

3 

16 

21 

34 

7 

14 

8 

20 

22 

45 

5 

12 

10 

33 

23 

34 

8 

6 

14 

28 

24 

31 

1 

12 

2 

19 

25 

45 

4 

15 

10 

30 

es  eldiT 

Tj'idr r' r;-ri 
• - - • • 

dn.3?i  tc  wxooB  faewod  baa  tfeerigiH 


■ 

■ ; oi  nr" ' 

11,  • ■ 

c ■ • '•'•;  • 

■ ■ 

01 

ri 

I 

a ,8 

XX 

i » 

04^ 

22 

OX 

■i* 

r£ 

01 

4-2 

l . 

8* 

5 

£2 

21 

YX 

^X 

r r 

Y 

TS 

8 

XI 

d 

8 

21 

e 

cl 

Y 

ox 

I , 

7 «c 

XX 

XX 

Of 

SI 

SI 

* r . 

£ r 

a 

. r> 

01 

XX 

11 

x£ 

21 

l 

1 

£2 

a 

X4 

'1 

. 

M 

I 

PI 

IS 

T£ 

OS 

Y 

IS 

01 

21 

3^ 

22 

*x 

AC 

£2 

gl 

21 

I 

X3 

X2 

rt 

£ 

cS 

Table  30 


Industry 
Judge  B 

Highest  and  Lowest  Trait  Scores  of  Each  Subject 


Sub- 

Trait 

Trait 

jects 

Hi*h 

Number 

Low 

Number 

Spread 

1 

50 

9 

15 

10 

35 

2 

37 

7 

15 

13 

22 

3 

40 

10 

25 

h 2* 12 
10 

15 

4 

35 

4,  9 

10 

25 

5 

43 

13 

25 

n 

18 

6 

45 

3 

10 

n 

35 

7 

35 

7,  9, 
11-14 

20 

8 

15 

8 

35 

4,  7,  13 

15 

8 

20 

9 

35 

7,  9 

25 

1-3,  5,  10 
14,  15 

10 

10 

45 

7 

15 

2 

30 

11 

35 

7 

12 

10 

23 

12 

40 

7 

15 

8 

25 

13 

35 

4 

10 

8,  9,  11 

25 

14 

40 

8 

12 

10 

28 

15 

35 

3.  5,  6 

15 

10,  13 

20 

16 

30 

4 

10 

3 

20 

17 

40 

2,  9,  13 

25 

5,  6,  8 

11,  14 

15 

18 

35 

13 

15 

14 

20 

19 

37 

8 

15 

1 

22 

20 

40 

8 

20 

1 

20 

21 

33 

12 

15 

8 

18 

22 

40 

3,  9 

15 

10,  13 

25 

23 

40 

8 

10 

14 

30 

24 

28 

4 

10 

11 

18 
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20 

11 
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Table  31 

Industry 
Judge  G 

Highest  and  Lowest  Trait  Scores  of  Each  Subject 


Sub- 

jects 

High 

Trait 

Number 

Low 

Tra  it 
Number 

Spread 

1 

45 

8,  9,  12 

18 

10 

27 

2 

35 

1,  7 

10 

8 

25 

3 

35 

5,  10,  13 

22 

14 

13 

4 

35 

4,  8,  9 

6 

10 

29 

5 

46 

12 

25 

2,  6,  8 

21 

6 

40 

5 

10 

9 

30 

7 

42 

14 

20 

4 

22 

8 

38 

7 

10 

8 

28 

9 

32 

1 

18 

10 

14 

10 

45 

11 

15 

3,  15 

30 

11 

33 

7 

12 

13 

21 

12 

35 

7 

10 

8 

25 

13 

32 

2 

6 

11 

26 

14 

35 

3,  5,  6 

15 

10,  13 

20 

8,  15 

15 

40 

5 

18 

10,  13 

22 

16 

30 

1,  6,  10 

10 

8 

20 

15 

17 

43 

13 

20 

8 

23 

18 

39 

6 

20 

14 

19 

19 

42 

2 

23 

1 

19 

20 

35 

4,  8 

15 

3 

20 

21 

35 

12,  13 

10 

8 

25 

22 

43 

5 

18 

13 

25 

23 

37 

15 

15 

x4  7*  11 

22 

24 

33 

1 

15 

8 

18 

25 

40 

4 

25 

5,  8-10 

15 
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Table  32 


Nurses 
Judge  A 


Highest 

and 

Lowest  Trait 

Scores 

i of  Each 

Subject 

Sub- 

Trait 

Trait 

jects 

High 

Number 

Low 

Number 

Spres 

1 

21 

2,  12 

5 

1,  3 

16 

2 

25 

4,  5,  7 

5 

2 

20 

9 

3 

40 

12 

25 

1 

15 

4 

35 

2 

17 

14 

18 

5 

45 

2,  3,  5 

24 

1 

21 

15 

6 

39 

13 

3 

14 

36 

7 

41 

13 

24 

2,  11 

17 

8 

45 

1,  3 

28 

15 

17 

9 

26 

7 

10 

8 

16 

10 

45 

5 

22 

2 

23 

11 

32 

15 

19 

13 

13 

12 

28 

15 

5 

7 

23 

13 

20 

12 

5 

1 

15 

14 

42 

6 

21 

13 

21 

15 

33 

15 

15 

1 

18 

16 

28 

13 

10 

1 

18 

17 

35 

14 

20 

2,  9 

15 

18 

25 

1,  2,  4 

10 

7,  9 

15 

10,  12,  15 

19 

40 

5,  8,  9 

25 

1,  4,  11 

15 

20 

35 

5 

15 

7,  11,  12 

! 20 

14 

21 

35 

13 

15 

1,  3,  7 

20 

11 

22 

30 

5,  6 

5 

12 

25 

23 

25 

2,  5,  15 

14 

8 

11 

24 

30 

5 

15 

1 

15 

25 

25 

12 

5 

11 

20 
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Table  33 


Nurses 
Judge  B 

Highest  and  Lowest  Trait  Scores  of  Each  Subject 


Sub- 

jects 

High 

Trait 

Number 

Low 

Trait 

Number 

Spread 

1 

24 

12 

8 

1 

16 

2 

30 

8 

15 

2,  13 

15 

3 

42 

6,  8 

25 

11 

17 

4 

35 

10 

15 

14 

20 

5 

40 

6,  8 

15 

10 

25 

6 

25 

1,  4,  10 
12,  13 

5 

8,  14 

20 

7 

41 

15 

25 

7,  11,  14 

16 

8 

45 

1,  12 
13,  14 

25 

6 

20 

9 

37 

8 

15 

1 

22 

10 

39 

5,  15 

24 

2,  3 

15 

11 

45 

7 

21 

2 

24 

12 

29 

10,  13 

10 

7 

19 

13 

25 

11 

5 

1 

20 

14 

40 

8 

25 

10,  13 

15 

15 

25 

12 

11 

1 

14 

16 

30 

13,  15 

9 

14 

21 

17 

40 

7,  8,  14 

25 

4,  11,  12 

15 

18 

29 

2 

5 

7 

24 

19 

45 

14 

22 

4 

23 

20 

25 

4,  5,  10 
13 

9 

11 

16 

21 

40 

13 

10 

5 

30 

22 

30 

4 

10 

13 

20 

23 

35 

2,  3,  6 
11,  12,  14 
15 

25 

1,  4,  8 
10 

10 

24 

27 

15 

10 

2 

17 

25 

20 

9,  12 

5 

3,  7,  15 

15 

Table  34 


Nurses 
Judge  C 

Highest  and  Lowest  Trait  Scores  of  Each  Subject 


Sub- 

jects 

High 

Trait 

Number 

Low 

Trait 

Number 

Spread 

1 

25 

12,  13 

2 

15 

23 

2 

29 

9,  15 

10 

2 

19 

3 

41 

10 

25 

13 

16 

4 

30 

10,  13 

11 

14 

19 

5 

40 

5,  8,  13 
15 

19 

1 

21 

6 

25 

1,  2,  4 

5 

9 

20 

7 

37 

10 

24 

7 

13 

8 

40  ' 

h 7’ 13 

25 

4 

15 

9 

35 

11 

2 

24 

10 

41 

5 

14 

2 

27 

11 

40 

11 

16 

2 

24 

12 

31 

3,  11 

8 

7 

23 

13 

25 

12 

10 

2,  10 

15 

14 

35 

3,  14 

25 

1,  7 

10 

15 

25 

3,  7,  10 
12 

9 

1 

16 

16 

33 

13 

10 

14 

23 

17 

35 

1,  7 

20 

9 

15 

18 

28 

2 

10 

7 

18 

19 

45 

7,  14 

22 

2 

23 

20 

25 

5 

5 

7,  14 

20 

21 

35 

12,  13 

15 

2,  11 

20 

22 

35 

4,  6,  9 

10 

13 

25 

23 

40 

15 

24 

1 

16 

24 

25 

4,  5,  8 
10,  15 

5 

3 

20 

25 

25 

13,  15 

5 

7 

20 

Table  35 


Boyfs  Club 
Judge  A 

Range  of  Scores 


Sub- 

jects 

High 

Score 

Low 

Score 

Spread 

Average 

Spread 

Those 

Subjects  Who  Scored  Highest 

On  At  Least 

One  Trait 

9 

44 

26 

IB 

12 

45 

25 

20 

13 

46 

24 

22 

19 

49 

16 

33 

21 

41 

24 

17 

24 

45 

15 

30 

25 

49 

IB 

31 

24.43 

Those  Subjects  Who  Scored 

Lowest 

On  At  Least 

One  Trait 

1 

42 

10 

32 

16 

39 

4 

35 

17 

19 

4 

15 

IB 

18 

B 

10 

20 

31 

5 

26 

23.60 

Other  Subjects 

2 

37 

10 

27 

3 

40 

19 

21 

4 

40 

21 

19 

5 

35 

15 

20 

6 

39 

14 

25 

7 

40 

15 

25 

B 

39 

19 

20 

10 

43 

24 

19 

11 

28 

7 

21 

14 

42 

21 

21 

15 

44 

25 

19 

22 

40 

13 

27 

23 

38 

15 

23 

22.08 
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Table  36 


Boy*s  Club 
Judge  B 

Range  of  Scores 


Sub- 

jects 

High 

Score 

Low 

Score 

Spread 

Average 

Spread 

Those 

Subjects  Who 

Scored 

Highest 

On  At  Least 

One  Trai 

t 

6 

50 

6 

44 

7 

50 

10 

40 

9 

50 

30 

20 

12 

50 

17 

33 

13 

50 

27 

23 

19 

50 

15 

35 

21 

50 

21 

29 

22 

50 

20 

30 

23 

50 

16 

34 

24 

50 

20 

30 

25 

49 

13 

36 

32.18 

Those  Subjects  Who  Scored 

Lowest 

On  At  Least 

One  Trait 

1 

49 

4 

45 

4 

49 

6 

43 

6 

50 

6 

44 

16 

40 

11 

29 

17 

25 

3 

22 

18 

30 

1 

29 

20 

40 

1 

39 

35.86 

Other  Subjects 

2 

36 

6 • 

30 

3 

40 

10 

30 

5 

36 

22 

14 

8 

40 

10 

30 

10 

46 

26 

20 

11 

36 

10 

26 

14 

40 

26 

14 

15 

48 

22 

26 

23.75 
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Table  37 


Boy1 s Club 
Judge  C 

Range  of  Scores 


Sub- 

jects 

High 

Score 

Low 

Score 

Spread 

Average 

Spread 

Those  Subjects 

Who  Scored  Highest 

On  At  Least  One  Trait 

3 

44 

11 

33 

9 

48 

29 

19 

10 

45 

19 

26 

12 

50 

25 

25 

13 

50 

27 

23 

14 

50 

10 

40 

15 

47 

20 

27 

19 

49 

21 

28 

21 

45 

21 

24 

22 

44 

26 

18 

26*30 

Those 

Subjects 

Who  Scored  Lowest 

On  At  Least  One  Trait 

1 

34 

13 

21 

6 

40 

4 

36 

7 

44 

12 

32 

11 

29 

1 

28 

17 

27 

10 

17 

18 

21 

11 

10 

24 

44 

7 

37 

25.  86 

Other 

Subjects 

2 

37 

16 

21 

4 

41 

16 

25 

5 

38 

18 

20 

8 

35 

19 

16 

• 

16 

43 

7 

36 

20 

30 

14 

16 

23 

43 

17 

26 

25 

42 

23 

19 

22.38 
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Table 

High  School 
Judge 
Range  of 

38 

Students 

A 

Scores 

Sub- 

High 

Low 

Average 

jects 

Score 

Score 

Spread 

Spread 

Those  Subjects 

Who 

Scored  Highest 

On  At  Least 

One  Trait 

2 

50 

34 

16 

7 

40 

30 

10 

11 

50 

40 

10 

17 

50 

40 

10 

11.50 

Those  Subjects  Who  Scored 

Lowest 

On  At  Least 

One  Trait 

3 

40 

5 

35 

5 

20 

5 

15 

12 

35 

5 

30 

13 

30 

10 

20 

20 

35 

15 

20 

24.00 

Other  Subjects 

1 

35 

15 

20 

4 

40 

25 

15 

6 

40 

20 

20 

8 

35 

15 

20 

9 

40 

20 

20 

10 

30 

15 

15 

14 

40 

10 

30 

15 

40 

25 

15 

16 

30 

20 

10 

18 

40 

20 

20 

19 

40 

30 

10 

21 

40 

30 

10 

22 

30 

20 

10 

23 

30 

15 

15 

24 
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Table  39 


Sub- 

.iects 


2 

11 

17 


3 

5 

8 

12 

13 

20 


1 

4 

6 

7 

9 

10 

14 

15 

16 
18 
19 
21 
22 

23 

24 

25 


High  School  Students 
Judge  B 

Range  of  Scores 


High 

Low 

Average 

Score 

Score 

Spread 

Spread 

Those 

Subjects  Who 

Scored  Highest 

On  At  Least 

One  Trait 

50 

35 

15 

50 

40 

10 

10.67 

50 

43 

7 

Those  Subjects  Who  Scored 

Lowest 

On  At  Least 

One  Trait 

40 

5 

35 

20 

5 

15 

23 

10 

13 

33 

3 

30 

32 

10 

22 

35 

13 

22 

22.83 

Other  Subjects 

35 

16 

19 

42 

25 

17 

42 

23 

19 

42 

30 

12 

39 

22 

17 

32 

15 

17 

40 

11 

29 

44 

25 

19 

35 

18 

17 

42 

22 

20 

40 

28 

12 

40 

28 

12 

30 

18 

12 

32 

18 

14 

35 

20 

15 

16.50 

30 

17 

13 
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Table  40 


High  School  Students 
Judge  C 

Range  of  Scores 


Sub-  High  Low  Average 

jeots  Score Score  Spread  Spread 

Those  Subjects  Who  Scored  Highest 
On  At  Least  One  Trait 


7 45  27  IS 

11  49  3S  11 

17  50  41  9 12.67 

Those  Subjects  Who  Scored  Lowest 
On  At  Least  One  Trait 


3 

40 

5 

20 

8 

38 

12 

32 

13 

31 

20 

31 

1 

31 

2 

40 

4 

40 

6 

40 

9 

45 

10 

25 

14 

40 

15 

45 

16 

36 

18 

41 

19 

40 

21 

40 

22 

31 

23 

35 

24 

26 

25 

36 

5 

35 

5 

15 

10 

28 

8 

24 

10 

21 

10 

21 

Other 

Subjects 

15 

16 

30 

10 

25 

15 

20 

20 

25 

20 

13 

12 

15 

25 

20 

25 

20 

16 

21 

20 
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10 

30 
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11 
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24 
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24.00 
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Table  41 


Industry 
Judge  A 

Range  of  Scores 


High  Low  Average 

Score Score  Spread  Spread 

Those  Subjects  Who  Scored  Highest 
On  At  Least  One  Trait 


48 

11 

37 

44 

22 

22 

48 

24 

24 

46 

17 

29 

46 

15 

31 

42 

12 

30 

45 

12 

33 

29.43 


Those  Subjects  Who  Scored  Lowest 
On  At  Least  One  Trait 


3 

17 

11 

9 

6 

7 

9 

6 


37 

29 

27 
22 

33 

34 
26 

28 


Other  Subjects 


40 

46 

38 
31 

39 

41 
35 
34 


40 

38 
37 

39 

41 
34 

44 
37 
34 
31 

45 


11 

14 

18 

9 

23 

14 

20 

21 

14 
12 

15 


29 
24 

19 

30 
18 

20 
24 
16 
20 
19 
30 
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Table  42 


Industry 
Judge  B 

Range  of  Scores 


Sub-  High  Low  Average 

.iects  Score Score  Spread  Spread 

Those  Subjects  Who  Scored  Highest 
On  At  Least  One  Trait 


1 

50 

15 

35 

3 

40 

25 

15 

5 

43 

25 

18 

6 

45 

10 

35 

10 

45 

15 

30 

17 

40 

25 

15 

25 

45 

20 

25  24.71 

Those  Subjects 

Who  Scored  Lowest 

On  At  Least  One  Trait 

2 

37 

15 

22 

4 

35 

10 

25 

6 

45 

10 

35 

11 

35 

12 

23 

13 

35 

10 

25 

15 

35 

15 

20 

16 

30 

10 

20 

19 

37 

15 

22 

21 

33 

15 

18 

22 

40 

15 

25 

23 

40 

10 

30 

24 

28 

10 

18  23.5S 

Other 

Subjects 

7 

35 

20 

15 

8 

35 

15 

20 

9 

35 

25 

10 

12 

40 

15 

25 

14 

40 

12 

28 

18 

35 

15 

20 

20 

40 

20 

20  19.71 
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Industry 
Judge  C 

Range  of  Scores 


Sub-  High  Low  Average 

.iects  Score Score  Spread  Spread 

Those  Subjects  Who  Scored  Highest 
On  At  Least  One  Trait 


1 

45 

18 

27 

5 

46 

25 

21 

7 

42 

20 

22 

10 

45 

15 

30 

17 

43 

20 

23 

19 

42 

23 

19 

22 

43 

18 

25 

25 

40 

25 

15  22.75 

Those  Subjects 

Who  Scored  Lowest 

On  At  Least  One  Trait 

2 

35 

10 

25 

4 

35 

6 

29 

8 

38 

10 

28 

10 

45 

15 

30 

11 

33 
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10 
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19 
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10 
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15 

22  24.17 

Other 

Subjects 
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Table  44 


Nurses 
Judge  A 

Range  of  Scores 


Sub-  High  Low  Average 

.jects  Score Score  Spread  Spread 

Those  Subjects  Who  Scored  Highest 
On  At  Least  One  Trait 


3 

40 

25 

15 

5 

45 

24 

21 

7 

41 

24 

17 

8 

45 

28 

17 

10 

45 

22 

23 

14 

42 

21 

21 

19 

40 

25 

15  18.43 

Those  Subjects  Who 

Scored  Lowest 

On  At  Least 

One  Trait 

1 

21 

5 

16 

2 

25 

5 

20 

6 

39 

3 

36 

12 

28 

5 

23 

13 

20 

5 

15 

22 

30 

5 

25 

23 

25 

14 

11 

25 

25 

5 

20  20.75 

Other  Subjects 

4 

35 

17 

18 

9 

26 

10 

16 

11 

32 

19 

13 

15 

33 

15 

18 

16 

28 

10 

18 

17 
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Table  45 


Nurses 
Judge  B 

Range  of  Scores 


Sub-  High  Low  Average 

.iects  Score Score  Spread  Spread 

Those  Subjects  Who  Scored  Highest 
On  At  Least  One  Trait 


3 

42 

25 

17 

7 

41 

25 

16 

8 

45 

25 

20 

11 

45 

21 

24 

19 

45 

22 

23 

23 

35 

25 

10  18.33 

Those 

Subjects  Who 

Scored  Lowest 

r 

On  At  Least 

One  Trait 

1 

24 

8 

16 

6 

25 

5 

20 

13 

25 

5 

20 

18 

29 

5 

24 

20 

25 

9 

16 

21 

40 

10 

30 

22 

30 

10 

20 

24 

27 

10 

17 

25 

20 

5 

15  19.78 

Other  Subjects 

2 

30 

15 

15 

4 

35 

15 

20 
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40 

15 

25 

9 

37 

15 

22 

10 

39 

24 

15 

12 

29 

10 

19 

14 
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15 
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11 

14 
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17 

40 
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Nurses 
Judge  C 

Range  of  Scores 


High  Low  Average 

Score Score  Spread  Spread 

Those  Subjects  Who  Scored  Highest 
On  At  Least  One  Trait 


41 

25 

16 

40 

19 

21 

40 

25 

15 

41 

14 

27 

40 

16 

24 

35 

25 

10 

35 

20 

15 

45 

22 

23 

40 

24 

16 

18.56 

Those 

Subjects 

Who  Scored 

Lowest 

On  At  Least  One  Trait 


2 

23 

10 

19 

5 

20 

10 

15 

9 

16 

5 

20 

10 

25 

5 

20 

5 

20 

Other  Subjects 


25 

29 

25 

25 

25 

25 

35 

25 

25 


30 
37 
35 

31 
33 
28 

35 


11 

24 

11 

8 

10 

10 

15 


19 
13 
24 
23 
23 
18 
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MANUAL  OF  INSTRUCTIONS 
FOR  THE 

VOCATIONAL  SERVICE  RATING  SCALE 


A WORD  OF  CAUTION 

Ratings  made  without  the  diligent  use  of  the  man- 
ual are  not  worth  one's  time  and  effort* 

In  order  to  use  this  rating  scale  to  full  advan- 
tage, the  judge  or  rater  must  carefully  read  and  fol- 
low the  Instructions  and  explanations  described  in  the 
rating  manual* 

Ratings  of  personality  traits  are  vital  and  in- 
teresting to  both  the  individual  rating  and  the  indi- 
vidual rated*  Maintain  a scientific  mode  of  approach. 
Rate  individuals  on  the  Vocational  Service  Rating  Scale 
ONLY  after  you  have  mastered  the  fundamental  techniques 
of  rating  procedures.  A selected  bibliography  is  list- 
ed for  those  who  wish  to  delve  further  into  the  fascina- 
ting field  of  personality  ratings. 

Pages  216  to  221  inclusive  are  of  utmost  impor- 


tance 
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Rating  of  people  is  not  new;  it  has  always  been 
a conscious  or  almost  unconscious  mental  process  of 
every  social  relationship.  Men  and  women,  boys  and 
girls,  are  constantly  observing  one  another,  and  from 
external  behavior  inferring  something  regarding  mental 
traits.  These  estimates  are  often  built  up  unconscious- 
ly, but  their  effect  is  cumulative.  When  an  individual 
is  asked  for  an  opinion  regarding  another,  he  may  then 
realize  he  has  already  formulated  an  opinion.  This 
opinion,  nevertheless,  is  of  doubtful  value,  especially 
in  the  form  in  which  it  is  most  frequently  available. 

If  one  asks  a person  what  he  thinks  of  his  employer, 
or  a teacher  of  his  student,  the  answer  usually  Involves 
some  indefinite  generalities;  that  he  is  a "good”  man 
or  a "good"  student,  or  a ’’poor"  man  or  a Mlazy"  stu- 
dent. These  terms  are  obviously  quite  relative  and 
convey  radically  dissimilar  impressions  to  different 
individuals.  Being  a good  student  in  the  estimation 
of  one  teacher  may  be  equivalent  to  mediocrity  in  the 
estimation  of  another.  General  impressions  of  this 
type  are  apt  to  be  very  subjective  and  reflect  preju- 
dice. If  the  rater  has  had  some  unfortunate  experience 
with  the  individual  such  as  some  single  Instance  of 
unreliability,  he  is  apt  to  attribute  this  quality  to 
the  individual’s  entire  personality.  Therefore,  through 
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a rating  scale  cne  attempts  to  eliminate  these  biases 
and  general  evaluations  and  obtain  the  estimates  in  a 
more  scientific  manner.  This  may  be  accomplished  by 
judging  the  traits  separately,  and  then  if  need  be, 
combining  these  evaluations  into  a final  rating.  Sep- 
arate consideration  of  the  traits  in  this  manner  ob- 
viates some  snap  judgments  and  insures  that  all  judges 
record  their  impressions  in  a more  systematic  and  uni- 
form plan.  Judges  are  somewhat  less  apt  to  disagree 
when  rating  traits  separately,  than  when  estimating 
the  individual  as  a whole.  If  there  is  a disagreement, 
it  is  possible  to  analyze  it  because  of  the  constant 
characteristics  of  the  evaluating  scale. 

Proof  that  rating  scales  play  a very  significant 
part  in  industry  is  found  in  a recent  report  of  the 
National  Industrial  Conference  Board,  stating  that  such 
scales  were  used  by  94  concerns  which  they  investiga- 
ted. These  concerns  employed  a total  of  618,127  peo- 
1/ 

pie  • 

Teachers,  recognizing  the  importance  of  the  "here 
and  now"  in  knowing  their  pupils  may,  in  using  ratings 
in  the  classroom,  show  how  these  same  ratings  are  essen- 
tial to  the  world  of  work.  The  objectives  of  ratings 

l/  Employee  Rating.  National  Industrial  Conference 
Board,  New  York,  1938,  Studies  in  Personnel  Policy,  no. 
8,  p.  3. 
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in  industry  will  be  briefly  summarized  because  of  their 
importance  in  the  workaday  world.  They  are  also  a means 
of  indicating  to  the  teacher  or  counselor  ideas  which 
may  be  transmitted  to  their  students.  Facts  such  as 
these  have  reality  and  significance  and  may  clearly 
show  the  students  the  necessity  for  improvement  along 
personality  lines  deemed  important.  Emphasis  along  these 
lines  may  also  present  the  school  to  the  employer  in 
a new  and  different  light;  not  Just  as  a school,  but 
as  a real  place  of  training  for  youth. 

Six  main  objectives  of  ratings  in  industry  are 
the  following:  (1)  to  furnish  the  employee  with  some- 
thing definite  on  which  to  base  his  future  effort,  and 
at  the  same  time  to  provide  a stimulus  for  self-improve- 
ment; (2)  to  Indicate  the  need  of  transferring  employees 
when  they  are  found  to  be  unsuited  to  their  Jobs;  (3) 
to  establish  a fair  and  intelligent  policy  of  promo- 
tion; (4)  to  form  a basis  for  warning  employees  who 
are  not  adjusting  satisfactorily  to  their  work,  and/or 
aiding  them  in  improving  themselves;  (5)  to  operate  as 
an  effective  check  on  the  hiring  and  proper  placement 
of  employees;  (6)  to  develop  morale  and  confidence  in 
the  fairness  of  the  management  since  employees  will 
realize  their  work  and  attitudes  are  under  constant 
but  Impartial  scrutiny.  They  should  feel  assured  that 
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advancement  will  be  based  on  demonstrated  merit  and 
qualifications  and  not  on  favoritism  or  influence. 

Far  too  many  teachers  still  base  a pupil's  school 
performance  entirely  on  the  amount  of  subject  matter 
he  acquires  and  how  thoroughly  he  masters  it.  It  is 
frequently  overlooked,  or  not  entirely  recognized  that 
success  in  school  work  is  dependent  to  a great  extent 
on  school  habits.  The  attitudes  developed  by  the  pupil 
in  mastering  the  curriculum  are  often  the  ones  which  he 
will  use  in  making  adjustments  after  leaving  school. 

Since  it  is  realized  that  these  attitudes  determine 
success  in  school  work,  it  is  advisable  that  proper 
ones  be  formed  early.  As  an  aid  in  determining  whether 
desirable  traits  are  being  developed,  the  rating  scale 
may  be  of  great  significance. 

Vocational  guidance  is  recognized  today  as  an 
essential  function  of  secondary  education.  The  writer, 
in  speaking  to  hundreds  of  employers  in  various  fields, 
finds  that  employers  are  much  more  interested  in  character 
traits  such  as  initiative,  reliability,  and  alertness 
than  in  the  curriculum  studied  or  the  number  of  "A" 
grades  received.  This  writer  has  also  found  in  voca- 
tional placement  that  discharge  of  employees  for  lack 
of  ability  has  been  less  than  five  per  cent.  Reasons 
for  discharge  have  more  frequently  been  lack  of  reliabili- 
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ty,  untidy  appearance,  emotional  instability,  lack  of 
cooperation,  inaccurate  work,  and  inability  to  get  along 
with  fellow  employees  and  employer*  A study  of  the 
reasons  for  the  discharge  of  4,000  office  employees 
from  76  large  business  concerns  brought  out  the  fact 
that  in  90  per  cent  of  the  cases  deficiency  in  character 
and  personality  traits  were  the  important  factors. 

Only  ten  per  cent  were  fired  because  of  the  lack  of 

1/ 

specific  skills* 

Youth  must  be  prepared  for  the  day  when  he  leaves 
formal  schooling  and  steps  out  into  the  world  of  work. 
Teachers  can  tell  more  or  less  accurately  on  the  basis 
of  past  scholastic  records  what  to  expect  from  various 
pupils  in  mathematics,  history,  or  foreign  languages. 

In  diagnosing  the  individual  along  vocational  lines  or 
in  maladjustment  in  school,  achievement  and  Intelligence 
tests  or  grades  are  not  sufficient. 

Most  personality  tests  are  not  yet  reliable  or 
valid  enough  for  practical  use  except  in  the  hands  of 
a trained  person.  A satisfactory  rating  scale  is  valid 
and  reliable  and  may  be  skillfully  handled  by  the  teach- 
er who  is  more  Interested  in  helping  the  boys  and  girls 
than  she  is  in  teaching  subject  matter.  Teachers  can 

1/  H.  Hepner.  Psychology  Applied  to  Life  and  Work. 
Prentice-Hall.  New  York,  1941,  p.  9. 
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learn  how  to  rate  their  students  and  should  be  able  to 
do  this  just  as  readily  as  they  should  be  able  to  ad- 
minister achievement  tests  in  their  own  fields.  Teach- 
ers should  note  the  improvement  of  youth  along  person- 
ality lines,  just  as  they  test  their  students  education- 
ally to  determine  their  progress. 

Rating  scales  in  themselves  are  interesting.  Peo- 
ple as  a rule  are  resistant  to  psychological  measures 
that  they  do  not  understand  or  that  they  may  feel  con- 
tain "trick11  questions.  Answers  are  given  in  the  sup- 
posedly correct  manner.  Rating  scales  seem  to  override 
this  objection.  Ratings  are  a two  way  road;  an  indi- 
vidual may  be  rated,  and  he  may  rate  himself.  This 
rating  scale  is  interested  primarily  in  the  first  half 
of  the  "two  way  road"  without  minimizing  the  other  half. 
An  individual  may  rate  himself  high  or  low  and  be 
theoretically  right  or  wrong  or  approximately  accurate, 
but  the  important  fact  is  what  others  think  of  him. 

This  determines  to  a great  extent  his  adjustment.  The 
writer  has  found  this  to  be  succintly  brought  out  in 
the  use  of  the  rating  scale  in  the  club  situation. 

Gerald  believed  that  he  was  above  average  in  appearance, 
well  poised,  met  others  half-way,  and  had  many  likeable 
qualities.  In  reality  Gerald  had  rated  himself  much 
too  highly  and  in  fact,  kept  people  at  a distance  social- 
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ly«  Although  he  was  a member  of  a club,  he  was  on  the 
outside  rather  than  on  the  inside.  Discussing  each 
trait  in  detail,  we  were  able  to  improve  the  client’s 
total  behavior.  For  the  trait  of  appearance  it  is  com- 
paratively easy  to  mention  specific  instances  where 
improvement  may  be  made.  Loosened  buttons,  dirty  shirt, 
unshined  shoes;  they  are  all  different  faults  in  appear- 
ance, but  they  have  in  common  the  tendency  to  create 
an  unfavorable  impression. 

This  procedure  is  not  as  easy  for  such  a trait  as 
"reliability”,  but  nevertheless,  here  too,  the  obstacles 
may  be  hurdled.  For  example,  specific  instances  may  be 
shown  where  the  individual  has  disappointed  someone 
after  being  impressed  with  the  need  for  dependability. 

In  one  instance  it  may  mean  that  one  pupil  has  forgotten 
to  do  his  homework;  while  another  student  failed  to 
write  a note  to  a classmate  as  he  had  promised. 

They  are  both  different  situations.  The  responses  are 
different  in  action  as  well  as  degree,  but  fundamentally 
both  instances  show  lack  of  reliability.  It  is  this 
fact  that  interests  us. 

In  our  schools  we  teach  a great  many  specific  things, 
but  we  assume,  and  rightfully  so,  that  there  will  be 
carry-over.  Education  teaches  us  the  ability  to  genera- 
lize. We  expect  a reliable  person  to  be  reliable  not 
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only  when  keeping  a luncheon  engagement,  but  also  in 
fulfilling  promises  and  obligations  of  much  greater 
import*  As  a general  rule,  the  individual  who  is  not 
reliable  in  certain  situations  carries  this  behavior 
over  into  other  activities.  A pupil,  who  is  not  neat 
in  his  work  in  History,  will  probably  do  sloppy  work 
in  English.  This,  of  course,  can  not  be  accepted  as 
a dogmatic  truism*  However,  habits  are  ingrained  by 
repetition.  Once  a particular  action  of  form  of  behav- 
ior is  completed,  a repeated  response  is  easier.  Most 
individuals  build  up  patterns  of  behavior  within  them- 
selves; they  may  not  be  recognized  as  such  consciously, 
but  they  are  with  us  just  the  same.  Nevertheless,  if 
habits  could  not  be  changed,  teachers  may  as  well  go 
home,  and  the  schools  may  as  well  close  their  doors. 
Improvements  are  necessary  and  may  be  obtained  if  teach- 
ers are  able  to  see  the  individual  in  the  various  im- 
portant manifestations  of  his  personality. 

Traits  included  in  the  Vocational  Service  Rating 
Scale  were  selected  on  the  basis  of  the  opinions  of 
20  guidance  counselors,  opinions  of  475  employers  in 
giving  personality  qualifications  for  job  openings 
and  reasons  for  discharge,  personal  examination  by  the 
writer  of  over  100  rating  scales  used  in  school  or  in- 
dustry, and  traits  mentioned  in  the  literature  as  capable 
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of  measurement.  All  are  traits  which  influence  voca- 
tional success.  There  are  15  traits  in  the  entire  Scale. 
In  the  application  and  use  of  the  Rating  Scale,  one 
must  recognize  that  such  a rating  scale  can  never  be 
entirely  adequate.  If  one  were  to  appraise  scientific 
apparatus  built  for  a particular  purpose,  it  would  only 
be  necessary  to  determine  how  closely  it  measured  up 
to  the  original  specification.  We  have  exact  instruments 
and  controlled  methods  of  observation  available  for 
this  purpose.  We  have  few  such  instruments  applicable 
to  human  beings  and  none  in  the  field  of  personality 
measurement.  It  is  impossible  to  devise  a scale  so 
exact  that  an  individual  when  appraised  by  different 
judges  would  always  be  rated  the  same.  It  is,  there- 
fore, necessary  that  one  carefully  observe  and  study 
each  individual  from  as  many  facets  as  possible  and  fin- 
ally use  his  best  Judgment  in  determining  the  most 
equitable  rating  in  order  to  reduce  avoidable  errors 
to  a minimum. 

1/ 

W.  A.  McCall,  in  his  book,  Measurement . mentions 
the  following. 

"...There  are  few  if  any  measurements  made  by  a 
science  that  are  more  accurate  than  ratings  on 
personality  traits ... .But  it  is  possible  that  how 
much  intelligence  people  think  an  Individual  has 


1/  W.  A.  McCall,  Measurement . Macmillan  Co.  New  York, 
1939,  p.  311. 
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is  of  greater  moment  than  how  much  he  really  has. 

In  this  vitally  significant  area,  ratings  are 
delicately  accurate. 

’’They  are  even  more  significant  in  the  case 
of  personality  traits  that  have  no  existence  out- 
side of  the  mind  of  the  rater.  Thus,  an  individual’s 
force,  kindliness,  tact,  and  beauty  are  in  very 
essence  a matter  of  the  subjective  impression  created 
on  others • 

"So  in  this  sense,  and  it  is  a very  important 
sense,  subjective  measurement  is  exceedingly  accur- 
ate ....  l/ 

The  descriptive  phrases  on  the  Vocational  Service 
Rating  Scale  for  the  various  traits  are  necessarily 
brief  on  account  of  space  limitations.  Once  one  becomes 
familiar  with  the  Scale  and  the  various  characteristics 
which  may  be  grouped  under  a given  classification,  the 
descriptions  will  be  found  sufficient. 


Description  of  Plan  and  Basic  Attributes 
For  the  purpose  of  clarifying  the  procedure  and 
enabling  the  rater  to  make  a finer  discrimination  be- 
tween the  various  traits  and  attributes,  the  following 
section  will  be  devoted  to  definitions  and  descriptions. 
Meanings  are  additions  to  and  abstractions  from  defini- 
tions compiled  and  presented  in  Howard  C.  Warren's 

2/ 

Dictionary  of  Psychology  and  Webster's  New  Internation- 
al Ibid. 

2/  H.  C.  Warren,  Dictionary  of  Psychology.  Houghton 
Mifflin.  Boston,  1934,  x 372  pp. 


1/ 

al  Dictionary  of  the  English  Language ♦ In  the  use 
of  the  Rating  Scale,  follow  the  definitions  given  as 
your  guide  even  though  you  may  believe  others  may  be 
more  appropriate  or  satisfactory.  This  will  make  for 
standardized  meanings  for  all  raters  and  will  aid  in 
refinement  of  this  rating  scale. 

APPEARANCE 

Appearance  is  the  general  features  or  distinctive 
works  of  an  object  or  being,  as  noted  in  visual  obser- 
vation. It  is  the  state  in  which  a person  or  thing 
appears  without  implication  as  to  actual  or  inner  state. 
VOICE  AND  SPEECH 

Voice  equals  the  sounds  produced  by  the  vocal  or- 
gans in  which  any  succession  of  sounds  primarily  involves 
the  use  of  the  larynx.  The  sound  uttered  is  considered 
as  possessing  some  special  quality  or  character. 

Speech  is  a system  of  communication  through  conven- 
tional vocal  symbols.  The  faculty  of  expressing  thoughts 
by  words  or  articulate  sounds  is  the  power  of  speaking. 
ABILITY  TO  PRESENT  IDEAS 

Ability  equals  the  power  to  perform  responsive 

acts  Including  complex  coordinated  movements  and  solution 

of  mental  problems.  In  relation  to  ideas,  it  is  a psy- 

v Webster^s  New  International  Dictionary  of  the  English 
Language . Second  edition  unabridged.  G & C Merriam 
Company.  Springfield,  Massachusetts,  194-1,  xcvi  3210  pp. 
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etiological  process  having  a symbolic  or  representative 
rather  than  a direct  function.  In  simple  terms,  the 
ability  to  present  ideas  is  a product  of  reflection  or 
mental  concentration;  a formulated  thought  or  opinion. 
PHYSICAL  VITALITY 

Physical  vitality  is  the  ability  of  an  organism 
to  maintain  its  organic  existence  energetically  and 
function  full  of  life  and  vigor. 

INTELLIGENCE 

Intelligence  is  the  ability  to  meet  new  situations 
quickly  and  successfully.  It  is  measured  success  in 
performance  at  tasks  commonly  called  intellectual;  l.e., 
arithmetical  computations  or  completing  verbal  relations 
etc.  It  is  the  capacity  for  knowledge  and  understand- 
ing especially  as  applied  to  the  handling  of  novel  situa 
tions.  In  the  vernacular,  we  are  interested  whether 
the  individual  "sees  the  point"  or  not.  It  must  be 
judged  often  from  the  extent  of  the  Individual's  success 
in  grasping  general  ideas  and  adapting  them  to  specific 
situations . 

ALERTNESS 

Alertness  is  the  transition  from  a state  of  rela- 
tive passivity  to  an  active  one;  that  is  the  self  as 
alert.  It  is  the  preference  for  speed  showing  Itself 
as  an  Individual  difference  significant  in  tasks  which 
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offer  little  intellectual  difficulty  but  are  performed 
under  speed  conditions* 

ACCURACY 

Accuracy  is  the  absence  (or  relative  absence)  of 
error.  In  behavior  it  is  measured  by  the  degree  in 
which  the  action  conforms  to  the  given  conditions,  i.e., 
to  the  requirements  of  the  task. 

LEADERSHIP 

Leadership  is  the  role  of  one  Individual  as  ini- 
tiator, director,  or  organizer  of  group  activities  in 
a community  or  herd.  Leadership  depends  on  attitudes 
of  dominance.  Leadership  demands  the  ability  to  analyze, 
to  plan,  and  to  systematize. 

INITIATIVE 

Initiative  is  the  ability  or  capacity  for  original 
conception  and  independent  action.  It  is  the  action 
of  an  Individual  in  starting  a chain  of  events. 

EMOTIONAL  STABILITY 

Emotional  stability  is  a type  of  temperament  or 
personality  characterized  by  an  absence  of  unusual  var- 
iations in  emotional  reactions  or  traits. 

JUDGMENT 

Judgment  includes  appreciation  of  a situation, 
comparison  (in  which  the  concept  of  greater  or  less  is 
applied),  and  evaluation  (in  which  reference  is  made 
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to  a series  of  standards  of  beauty,  right,  goodness, 
or  economic  worth).  It  is  the  power  of  arriving  at  a 
wise  decision  or  conclusion  on  the  basis  of  indications 
and  probabilities,  when  the  facts  are  not  clearly  as- 
certained. 

COOPERATION 

Cooperation  can  be  represented  by  the  united  effort 
of  individuals  in  a social  group.  It  is  to  act  or  to 
operate  jointly  with  another  or  others;  to  concur  in 
action,  effort,  or  effect.  Cooperation  implies  a will- 
ing voluntary  participation  on  the  part  of  an  individual. 
It  indicates  an  attitude  of  voluntary  helpfulness  rather 
than  mere  participation. 

SOCIABILITY 

Sociability  is  the  disposition  to  companionship 
or  association  with  other  people. 

RELIABILITY 

Reliability  means  trustworthiness.  It  is  the  sug- 
gestion of  qualities  that  can  be  relied  or  depended 
upon.  An  individual  who  is  reliable  can  be  counted  on 
to  do  what  he  says  he  will  do. 

SELF-CONFIDENCE 

Self-confidence  implies  confidence  in  one's  own 
strength  or  powers  with  the  individual  regarded  as  con- 
scious of  his  own  continuing  identity  and  of  his  rela- 
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tion  to  the  environment.  It  Is  assurance  beyond  reason- 
able doubt. 

Discussion  of  Reliability  and  Validity 

Any  good  psychological  test  or  measuring  scale 
must  meet  the  test  of  reliability  and  validity.  Rating 
scales  have  long  been  used,  but  comparatively  little 
attention  has  been  given  to  the  question  of  reliability 
and  validity.  In  many  instances,  reliability  coeffi- 
cients have  not  been  mentioned,  and  in  many  instances 
where  they  have  been  reported,  results  have  been  dis- 
appointingly low  and  based  on  an  insufficient  number  of 
judges.  In  nearly  all  cases  directions  and  explanations 
of  rating  scales  have  been  grossly  inadequate. 

Evidence  in  the  literature  has  convincingly  shown 
the  need  of  at  least  three  judges  in  rating  an  individual 
in  order  to  maintain  sufficient  reliability.  Little 
research  in  the  literature  mentions  research  based  on  a 
sufficient  number  of  individuals  rated  by  three  differ- 
ent judges.  The  writer  has  tried  to  overcome  these 
objections.  Before  describing  techniques  employed, 
it  will  be  helpful  to  discuss  the  validity  of  rating 
scales.  Some  investigators  have  long  regarded  the 
attainment  of  validity  for  a rating  scale  as  almost 
impossible  because  no  criterion  is  available. 
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In  a psychological  test  of  adjustment,  if  the  scale 
differentiates  significantly  between  individuals  who 
are  delinquent  and  those  who  are  adjusted  in  the  communi- 
ty, one  may  say  that  the  test  measures  what  it  purports 
to  measure,  namely  adjustment,  and  it  is  valid.  In 
rating  scales  some  men  who  have  done  a great  deal  of 
research  such  as  McCall,  Remmers,  Bradshaw,  and  others 
believe  that  the  problem  of  validity  of  judgments  is 
hardly  pertinent. 

While  reliability  may  be  defined  as  the  accur- 
acy with  which  a measuring  instrument  measures 
whatever  it  does  measure,  validity  is  defined  as 
the  extent  to  which  the  instrument  measures  what 
it  purports  to  measure.  Since  it  is  student  judg- 
ments that  constitute  the  criteria,  reliability  and 
validity  are  in  this  case  synonymous,  l/ 

In  the  use  of  a rating  scale  for  vocational  service, 
it  is  the  judgment  of  employers  and  vocational  counselors 
that  constitutes  the  criteria,  and  reliability  and  va- 
lidity are  in  this  case  also  synonymous.  If  employers 
and  counselors  regard  these  traits  as  essential  for  voca- 
tional adjustment,  and  these  traits  are  consistently 
rated  by  different  Judges,  the  test  is  valid.  It  makes 
little  difference  whether  the  trait  we  call  alertness 
actually  is  alertness  or  quality  "x" . If  employers 
believe  that  this  trait  is  necessary  and  if  different 

l/  K.  H.  Remmers , "Reliability  and  Halo  Effect  of  High 
School  and  College  Students*  Judgments  of  Their  Teach- 
ers", Journal  of  Applied  Psychology:  1934s  18:  621. 
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judges  rate  an  individual  approximately  the  same  on  this 
particular  trait,  the  test  measures  what  it  purports  to 
measure.  The  fact  that  we  give  the  trait  a name  makes 
for  eoual  standards  and  workability  of  the  scale. 

1/ 

As  pointed  out  by  Thurstone:  It  is  not  of  any 

scientific  concern  whether  the  trait  really  exists.  It 

is  sufficient  for  scientific  purposes  that  he  behaves 

as  though  a common  trait  were  operative." 

2/ 

As  stated  by  T.  L.  Kelley:  "if  competent  judges  ap- 

praise Individual  A as  being  so  much  better  than  individual 
B as  individual  B is  better  than  individual  C,  then  it  is 
so,  there  is  no  higher  authority  to  appeal  to." 

In  testing  the  reliability  of  the  first  edition  of 
the  Vocational  Service  Rating  Scale,  the  following  pro- 
cedure wa3  employed.  Six  interested  judges  independ- 
ently rated  twenty-seven  high  school  boys  whom  they 
knew  well  and  who  were  members  of  a club  in  a local 
settlement  house.  Oral  directions  were  given  on  the 
use  of  the  Rating  Scale,  with  a period  for  explanation 
and  discussion  of  rating  scales  in  general.  (The  pre- 
sent rules  and  explanations  are  much  more  elaborate). 

l/  L.  L.  Thurstone,  The  Reliability  and  Validity  of  Tests. 
Edward  Bros.  Ann  Arbor,  Michigan,  1932,  p.  102. 

2/  T.  L.  Kelley,  The  Influence  of  Nurture  upon  Indivi- 
dual Differences.  Macmillan  Co.  New  York,  1926 , 

p.  9. 
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The  Pearson  Product  Moment  correlation  between  the  aver- 
age ratings  of  Judges  A,  B,  C,  and  D,  E,  F,  were  taken 
for  the  fifteen  traits  comprising  the  Scale.  Results 
are  shown  in  Table  47* 

The  revised  edition  of  the  Rating  Scale  is  not 
radically  different  from  the  original;  it  is  an  attempt 
to  remedy  mistakes  and  provide  improvements  that  were 
not  apparent  until  the  Scale  was  used  extensively. 

General  Rules  and  Explanations 

There  are  fifteen  traits  in  this  graphic  rating 
scale;  each  one  of  which  has  a five  point  description. 
Read  all  the  descriptions  extending  from  zero  to  five 
EVEN  THOUGH  YOU  BELIEVE  YOU  KNOW  APPROXIMATELY  WHERE 
THE  INDIVIDUAL  FALLS.  Then  decide  as  well  as  you  can 
where  in  your  judgment  the  person  rated  belongs  on  each 
trait.  You  can  check  anywhere  along  the  line.  For 
example,  a check  mark  in  the  second  description  divi- 
sion on  a particular  trait,  but  toward  the  third  descrip- 
tion means  that  the  individual  rated  almost  reaches 
the  third  division.  A check  mark  toward  the  lower  end 
of  the  second  division  means  the  rated  individual  al- 
most reaches  the  first  division.  A check  in  the  center 
of  the  second  description  or  near  it  means  the  indivi- 
dual approximately  fits  the  description  given  for  this 
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Table  47 


Coefficient  of  Correlation  and  Probable  Error 
Between  Six  Judges  Rating  Twenty-Seven  High  School  Boys 


Traits 


Appearance 
Voice  and  Speech 
Ability  to  Present  Ideas 
Physical  Vitality 
Intelligence 

Alertness 
Accuracy 
Leadership 
Initiative 
Emotional  Stability 

J udgment 

Cooperation 

Sociability 

Reliability 

Self-Confidence 


Coefficient  Probable 

of  Correlation  Error 


78 

.05 

83 

.04 

54 

.09 

88 

.03 

87 

.03 

86 

.03 

85 

.04 

.85 

.04 

78 

.05 

74 

.06 

70 

.07 

,66 

.07 

.78 

.05 

.53 

.09 

,41 

.11 
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number.  If  you  do  not  have  any  basis  for  judgment  of 
a particular  trait,  make  a notation  on  the  line;  NO 
OPPORTUNITY  TO  OBSERVE.  To  guess  a rating  just  for 
the  sake  of  putting  something  down  is  unfair  both  to 
the  individual  rated  and  to  the  entire  rating  plan. 

Make  every  effort  to  observe  individuals  in  action  or 
behavior  that  exemplify  these  traits.  Initial  ratings 
may  be  made  in  pencil;  reratings  may  be  made  with  a 
red  pencil  to  show  Improvement,  a blue  pencil  for  a 
lower  rating.  The  numerical  values,  zero  to  five,  should 
at  no  time  be  confused  with  any  system  of  "scholastic 
grading"  with  which  you  may  be  familiar.  They  consti- 
tute merely  an  arbitrary  scale  upon  which  the  differ- 
ence between  pupils,  job  applicants,  or  individuals  in 
the  same  general  classification  may  be  shown. 

Rule  1. 

An  extended  period  of  observation  should  pre- 
cede any  rating.  Too  often  people  are  rated  in  school, 
Industry,  and  other  social  relations  without  enough 
objective  behavior  as  a guide.  Rating  may  be  improved 
with  practice. 

Rule  2. 

Acquaint  yourself  thoroughly  with  the  discus- 
sion and  description  of  traits  in  the  previous  section. 


This  is  essential 
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Rule  3 


The  attitude  of  the  rater  or  judge  should  be 
objective  and  impartial.  Rate  all  people  in  the  same 
general  classification  on  the  same  basis,  even  though 
someone  in  the  group  may  be  a particular  friend  or  one 
in  whom  you  have  a friendly  Interest.  One  tendency 
that  many  judges  have  in  common  is  to  rate  all  indivi- 
duals, whom  they  know,  above  average  in  certain  traits, 
usually  the  desirable  ones.  This  is  the  error  of  len- 
iency. No  attempt  should  be  made  to  cover  up  an  indi- 
vidual^ weak  points.  One  of  the  advantages  of  the 
rating  scale  is  to  bring  low  attributes  to  a higher 
plane.  Better  adjustments  are  nearly  always  possible. 
Conscious  prejudice  sometimes  is  involved  in  rating, 
but  the  more  frequent  difficulty  is  some  unintentional 
bias  due  to  some  pleasant  or  unpleasant  incident  in 
the  past  as  regards  the  particular  individual  to  be 
rated.  One  must  learn  to  look  at  the  individual  at 
arm's  length  and  estimate  objectively  and  impartially. 
One  can  not  expect  absolute  freedom  from  prejudice  and 
bias,  but  one  should  be  able  to  Justify  a particular 
rating  with  specific  reasons  and  incidents.  In  this 
connection,  it  may  prove  VEK£  advisable  to  use  the  blank 
or  reverse  side  of  the  Rating  Scale  as  a summary  sheet 
for  anecdotal  records.  All  that  is  needed  is  the  date. 
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incident,  and  comment.  This  may  prove  to  be  of  immeasur- 
able value  to  the  teacher,  counselor,  supervisor,  or 
foreman  in  assigning  accurate  ratings  for  the  various 
traits.  An  example  of  what  is  meant  is  found  below. 

John  H.  August  15,  1942.  Incident.  Success- 
fully convinced  home  room  class  during  a debate 
that  it  should  compete  in  the  inter-school  athletic 
contest.  His  ideas  were  presented  forceably,  logi- 
cally, and  clearly. 

James  B.  August  12,  1942.  Incident.  Left 
running  job  press  to  talk  to  fellow  worker  in  an- 
other section  of  the  shop  although  told  repeatedly 
by  foreman  of  the  need  for  constant  attention  to 
the  machine. 

All  ratings  that  extend  from  one- tenth  to  one  or 
from  four  to  five  should  be  corroborated  by  an  anecdotal 
record  on  the  reverse  side  of  the  Rating  Scale. 

Rule  4. 

Your  estimate  should  be  based  on  actual  rather 
than  expected  performance.  Base  your  ratings  on  per- 
formance within  the  past  year. 

Rule  5« 

Research  has  shown  that  better  ratings  are 
made  if  the  Judges  are  Interested  and  there  is  suffi- 
cient time.  Hurried  ratings  are  of  questionable  value. 
Rule  6. 

Judges  should  rate  all  individuals  who  are  to 
be  rated  for  one  particular  trait,  such  as  "appearance", 
before  going  on  to  rate  for  "voice  and  speech",  "ability 
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to  present  ideas",  "physical  vitality",  and  so  on  down 
the  list.  Stated  differently,  no  one  individual  should 
be  rated  for  all  fifteen  traits  at  one  time.  All  in- 
dividuals should  be  rated  for  one  trait  before  starting 
ratings  on  the  next  trait.  This  avoids  a shifting  stan- 
dard. When  all  are  rated  on  the  same  trait  at  the  same 
time,  the  standard  for  all  practical  purposes  is  more 
likely  to  remain  the  same. 

Rule  7 • 

It  is  relatively  easy  to  be  influenced  by 
unusual  incidents  that  are  not  really  representative 
of  the  individual's  characteristic  behavior.  In  rat- 
ing make  every  effort  to  consider  all  incidents  that 
are  characteristic  of  the  rates' s behavior. 

Rule  8. 

Avoid  the  error  of  central  tendency.  Trans- 
lated into  every  day  terms,  this  means  that  Judges  are 
inclined  to  hesitate  to  give  extreme  Judgments  and  so 
tend  to  displace  those  rated  in  the  direction  of  the 
average  of  the  group.  All  the  abilities  of  an  indivi- 
dual do  not  occur  in  identical  degrees. 

1/ 

Driver  in  a discussion  of  the  distribution  of 
results  for  various  traits  says, 

1/  R.  S.  Driver,  "A  Case  History  in  Merit  Rating" , Per- 
sonnel: 194-0:  16:  155-156. 
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♦..It  has  been  revealed  as  a result  of  psy- 
chological investigation,  however,  that  for  one 
individual  the  traits  are  probably  distributed  in 
the  shape  of  a normal  curve.  By  that  it  is  meant 
that  if  one  individual  is  considered  from  100  dif- 
ferent points  of  view,  such  as  how  fast  he  learns, 
how  accurate  he  is,  etc.,  and  the  results  care- 
fully plotted,  there  will  result  a curve  indica- 
ting that  the  individual  has  some  outstandingly 
good  traits,  some  equally  poor  traits,  and  that 
most  of  his  abilities  are  distributed  in  a middle 
level. ...  1/ 

Extremes  in  the  scale  should  be  used,  but  in  the 
everyday  group  they  should  not  be  too  numerous.  An 
extreme  rating  should  be  one  in  whose  accuracy  you  feel 
very  confident.  Individuals  may  fall  into  any  one  of 
the  five  descriptions  for  each  trait. 

Rule  9. 

Avoid  the  tendency  to  rate  all  items  high 
or  low  depending  upon  your  general  attitude  toward  the 
person  rated.  This  is  called  the  "halo  effect",  and  it 
is  not  easy  to  overcome.  The  rater  must  review  each 
trait  in  an  objective  manner  and  keep  in  mind  only  the 
particular  trait  at  hand.  Reread  the  definitions  of 
the  traits  given  earlier;  then  bring  to  mind  specific 
types  of  behavior  in  relation  to  a certain  trait.  Do 
not  be  biased  by  strong  personality  traits,  but  consi- 
der the  particular  attributes  independently.  It  is  a 
rare  individual  who  ranks  high  or  low  in  every  single 
1/  Ibid. 
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trait  on  the  Scale.  Proceed  cautiously  when  ratings 
seem  to  fall  definitely  very  high  or  very  low* 

Rule  10. 

Avoid  what  is  called  the  logical  error  in 
rating  individuals.  Do  not  give  similar  ratings  in 
traits  simply  because  they  seem  logically  related  to 
your  way  of  thinking*  Base  your  rating  solely  on  ob- 
servable behavior  that  is  applicable  to  the  particular 
trait. 

Rule  11. 

Avoid  the  tendency  to  overrate  members  of 
the  same  sex  as  compared  to  members  of  the  opposite 
sex.  Conscious  awareness  of  this  fact  will  help  com- 
pensate for  this  difficulty. 

Rule  12. 

Ratings  must  be  made  independently  although 
it  is  tempting  to  talk  them  over  with  others  who  are 
making  similar  ratings.  Research  has  demonstrated  that 
greater  validity  is  obtained  by  averaging  the  independent 
estimates  than  by  having  the  Judges  sit  together  as  a 
committee  and  making  Joint  estimates. 

Rule  13. 

Check  over  your  ratings  to  make  sure  that 
you  have  not  neglected  to  rate  an  Individual  for  a 
particular  trait.  Make  certain  that  you  have  signed 
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your  name  as  well  as  those  of  the  persons  rated.  If 
you  are  unable  to  rate  an  Individual  because  of  a lack 
of  evidence  on  any  particular  trait,  make  a notation 
on  the  line  which  the  trait  occupies;  Hno  opportunity 
to  observe." 

Rule  14. 

Ratings  or  measurements  of  any  kind  that  are 
not  used  for  a purpose  are  a waste  of  time.  It  is  most 
satisfactory  in  the  school  situation  if  three  different 
ratings  by  three  different  teachers  at  at  least  three 
different  times  are  compiled  for  each  pupil.  The  relia- 
bility of  particular  ratings  is  considerably  increased 
if  the  ratings  of  three  different  judges  are  averaged. 

The  first  rating  may  be  approximately  after  the  first 
month  or  so  of  school.  The  second  and  third  ratings 
may  well  be  taken  at  the  marking  periods. 

Discussion  of  personality  traits  should  be  welcome 
supplementary  material  to  the  teacher  when  she  discusses 
grades  and  school  in  general  with  her  students.  Rating 
scales  give  a clearer  picture  of  the  weaknesses  and 
assets  of  the  individual  and  may  be  a valuable  tool 
not  only  in  vocational  guidance  but  also  in  reconstructive 

1/ 

education.  As  pointed  out  by  Guilford  in  his  book, 

1 / J.  P.  Guilford,  Psychometric  Methods.  McGraw  Hill. 

New  York,  1935,  p.  2^5* 
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Psychometric  Methods:  "Without  any  doubt  rating  scale 


methods  have  made  their  place  secure  in  industrial  prac- 
tice and  in  the  educational  field •" 
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SCORE  SHEET 


Teachers,  vocational  guidance  counselors,  person- 
nel managers,  psychologists,  and  others  who  are  inter- 
ested in  furthering  the  refinement  of  this  rating  scale 
and  in  establishing  norms  for  their  own  particular  situa- 
tion, may  accomplish  this  by  the  following  simple  di- 
rections . 

For  practical  purposes  a check  mark  placed  any- 
where along  the  line  can  be  measured  exactly,  although 
in  reality  we  can  never  hope  for  an  exact  rating  of 
2.8,  or  3.2,  or  3.5,  or  1.6,  and  so  on.  The  scale  ex- 
tends from  zero  to  five  by  tenths.  Measure  each  in- 
dividual's score  exactly  for  each  trait.  You  will  have 
fifteen  scores  for  each  person  rated.  Then  take  the 
average  for  the  entire  group  for  each  one  of  the  traits. 
You  will  then  have  fifteen  average  group  ratings.  Let 
us  illustrate  the  procedure  with  a hypothetical  group 
of  five  ninth-grade  pupils.  A,  B,  C,  D,  and  E with 
scores  on  appearance  as  follows: 
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Group  Dlf- 


Trait 

Pupil 

Score 

Average 

ference 

Appearance 

A 

2.4 

3.5 

- 1.1 

Appearance 

B 

3.0 

3.5 

- .5 

Appearance 

C 

4.1 

3.5 

+ .6 

Appearance 

D 

4.5 

3.5 

+ 1.0 

Appearance 

E 

3.3 

3.5 

- .2 

17.3 

Average  Score: 

3.5 

Note : 

Carry  out 

figures 

to 

two  decimal 

places ; 

3.43  is  3 

.4;  3.55 

is 

3.6 

The  average  for  the  group  of  five  ninth-grade  pu- 
pils for  the  trait  appearance  is  3*5.  In  discussing 
specific  traits  with  individuals,  one  may  point  out  how 
each  student  or  job  applicant  compares  with  the  aver- 
age of  the  group  in  your  particular  situation.  The  in- 
dividuals* names  may  be  given  in  letters  or  some  other 
system  of  code  employed  so  that  each  individual  knows 
only  his  own  score  and  that  of  the  average  of  the  group. 
One  can  readily  note  those  individuals  who  are  above 
or  below  average  on  the  trait  of  appearance . The  same 
procedure  may  be  applied  to  the  other  fourteen  traits 
on  the  rating  scale.  Eventually  enough  scores  will 
be  tabulated  from  year  to  year  so  that  a teacher  may 
say  with  a certain  amount  of  assurance  what  the  "aver- 
age" boy's  or  girl's  score  should  be  on  the  fifteen 
traits.  Group  averages  will  not  be  tabulated  unless 
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the  school  population  has  radically  changed  or  for  rea- 
sons of  research. 

In  industry  critical  scores  may  eventually  be  tabula- 
ted. Personnel  managers  may  statistically  determine 
what  degree  of  each  trait  or  what  score  is  necessary 
for  adjustment  on  the  Job.  By  using  as  a criterion  the 
scores  of  successful  people  already  on  the  Job,  those 
employees  who  constantly  fall  below  can  be  dropped  or 
assigned  to  other  positions  which  do  not  require  such 
high  scores.  Ratings  available  beforehand  from  teach- 
ers may  also  eliminate  many  individuals  who  on  the  basis 
of  past  performance  undoubtedly  do  not  have  the  necessary 
personality  characteristics  to  enter  a particular  occupa- 
tion. 

If  three  Judges  rate  the  same  Individual  (and  three 
are  needed  for  high  reliability)  the  same  general  pro- 
cedure is  followed.  Let  us  again  illustrate  with  a 
hypothetical  group  of  ninth-grade  pupils:  F,  G , H,  I, 
and  J,  rated  by  three  Judges  and  whose  scores  are  as 
follows : 


Trait 

Pupil 

Pupil's  Scores 
Judges 

1 2 3 

Pupil' s 
Average 
Score 

• 4' 

Group 

Score 

Dif- 

ference 

Appearance 

F 

2.3 

2.5 

2.4 

2.4 

2.9 

- .5 

Appearance 

G 

3.5 

3.0 

2.5 

3.0 

2.9 

+ .1 

Appearance 

H 

4.1 

3.8 

2.5 

4.0 

2.9 

+ 1.1 

Appearance 

I 

2.4 

2.1 

2.9 

2.5 

2.9 

- .4 

Appearance 

J 

2.0 

2.5 

3.0 

2.5 

2.9 

- .4 
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The  writer  would  appreciate  your  sending  in  your 
average  group  score  ratings  for  each  trait  for  the  group 
that  you  have  rated.  No  matter  how  many  people  are  in 
your  particular  group,  you  need  send  in  only  fifteen 
average  group  scores. 

Please  fill  in  your  name,  position  in  detail,  num- 
ber in  group  you  have  rated,  designation  of  group  (for 
example,  tenth-grade  male  students),  and  the  major  en- 
vironment where  the  group  as  a whole  was  observed  such 
as  school,  industry,  etc.  Although  results  will  be 
used  in  the  refinement  of  the  Scale,  names  of  schools 
or  individuals  using  the  Scale  will  not  be  mentioned 
unless  so  requested. 

For  your  convenience,  a form  that  you  may  fill  in 
and  send  to  the  writer  is  on  the  next  page. 
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VOCATIONAL  SERVICE  RATING  SCALE  SCORE  SHEET 


NAME: 


POSITION: 


NUMBER  IN  GROUP:  PLACE  OF  OBSERVANCE 

FOR  GROUP: 


DESIGNATION  OF  GROUP: 


Trait  Group  Score 

Appearance  

Voice  and  Speech  

Ability  to  Present  Ideas  

Physical  Vitality  

Intelligence  

Alertness  

Accuracy  

Leadership  

Initiative  

Emotional  Stability  

Judgment  

Cooperation  

Sociability  

Reliability  

Self-confidence  
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VOCATIONAL  SERVICE  RATING  SCALE 


Person  Rated  _ 
Person  Rating 


M i dd  le 


Date  _ 


. Year_ 


INSTRUCTIONS:  Keeping  the  definition  of  the  trai 
minimum  and  maximum  by  placing  a check  mark  (vO 


in  mind,  rate  the  individual  between 
n the  proper  scale  at  the  point  where 
solely  on  evidence.  Try 


APPEARANCE 

not  to  omit  ratin 

MINIMUM  RATING 

1 1 t ■ 1 1 1 r 1 1 1 

gs  of  any  traits. 

1 1 1 1 1 i 1 1 1 l 

! 1 1 1 1 1 1 1 1 

1 

1 1 1 1 1 1 1 1 1 ! 

MAXIMUM  RATH 
J 1 1 1 1 1 1 1 1 ] 

0 Unsuitable,  | 

Creates  rather  2 

Suitable, 

3 

Creates  distinct-  4 

Impressive,  com- 

slovenly, 

unfavorable 

acceptable, 

ly  favorable 

mands  admiration 

repellent 

impress  ion 

appropriate 

impress  ion 

VOICE  AND 

1 1 ' 1 ' 1 1 1 1 1 1 

....  1 ...  . L 

1 1 1 1 1 1 1 1 1 

_L 

1 1 1 1 1 1 1 1 1 1 

1 1 1 1 | 1 1 1 l 

i| 

SPEECH 

0 Irritating  or  | 

Understandable  2 

Neither  conspic- 

3 

Definitely  q 

Exceptionally 

indist inct 

but  rather  un- 

uously  pleasant 

pleasant  and 

clear  and 

pleasant 

nor  unpleasant 

distinct 

pleas ing 

acceptable 

ABILITY  TO 

1 1 i 1 1 1 1 1 1 1 1 

1 1 1 1 1 1 1 1 1 1 

1 1 l 1 1 1 1 1 l 

1 

1 1 1 1 1 1 1 1 1 1 

1 1 1 1 1 1 1 1 1 

j 

PRESENT  IDEAS 

0 Confused,  il-  1 

Tends  to  become  2 

Usually  gets 

3 

Shows  superior  q 

Unusually  clear 

logical,  sloven- 

involved,  inexact  c 

his  ideas 

ability  to  ex- 

and  convincing 

ly  thinking 

scattered  presen- 

across  well 

press  himself, 

t.it  ion 

concise  and  to 

PHYSICAL 

1 1 1 1 1 1 1 1 1 1 1 

1 1 i 1 1 1 1 1 1 1 

1 1 1 1 1 1 1 i_j 

_L 

the  point 

■ 1 1 1 1 1 1 1 1 1 

1 1 1 1 1 1 1 1 1 

l 

VITALITY 

0 Little  physical  | 

Be  1 ow  average  2 

Suitable  or 

3 

Exhibits  superior  4 

Outstand ing 

5 

endurance 

in  physical 

acceptable 

physical  endur- 

phys ical 

vitality 

ance 

vitality 

INTELLIGENCE 

1 1 1 1 1 1 1 1 1 1 1 

1 1 1 1 1 1 > 1 1 1 

l 1 1 1 1 I 1 11 

1 

1 1 1 1 1 1 1 1 1 1 

1 1 1 1 1 1 1 1 1 

j 

0 Exceedingly  dull  . 

Not  too  bright  ^ 

Shows  average 

3 

Brighter  than  q 

Except ionally 

intelligence 

average 

intelligent 

ALERTNESS 


1 1 i 1 1 1 


LEADERSHIP 


INITIATIVE 


EMOTIONAL 

STABILITY 


JUDGMENT 


COOPERATION 


SOCIABILITY 


0 Slow  in  grasping 

| Slow  in  grasping  2 Nearly  always 

g Rather  quick  in  4 

Exceptionally  5 

the  obvious 

meaning  of  inter-  grasps  intent  of 

v iewer's  quest  ions  interviewer's 

grasping  ques- 
tions and  new 

keen  and  un- 
derstand ing 

1 1 1 1 1 1 1 1 1 1 

questions 

1 1 1 l 1 1 1 1 1 1 1 l 1 1 1 1 1 1 1 1 

facts 

1 1 1 1 . 1 1 1 1 1 1 Lx 

1 1 1 1 1 1 1 1 1 

0 Does  inexact 

1 Below  average 

2 Suitable  or  ac-  3 

Work  accuracy 

n Accomplishes  ex-  5 

work,  makes 

in  accuracy, 

ceptable,  oc- 

above  average 

ceedingly  exact 

many  errors 

careless 

casional  checking 

work 

1 1 1 1 1 1 1 1 1 

' 1 ' ''  1 1 1 1 i 

required 

1 1 1 1 1 1 1 1 1 1 1 1 

1 1 1 1 1 1 1 1 

1 1 1 1 1 1 1 1 1 1 1 1 

0 Prefers  follow- 

j Often  fails  to 

2 Occasionally 

3 Often  takes 

4 Plans  for  and  g 

ing  plans  of 

secure  support 

takes  leading 

leading  part 

directs  others 

others 

of  his  ideas 

part  in  group 

in  group 

successfully 

1 1 1 1 1 1 1 1 1 1 

l 1 1 1 > 1 1 1 1 

activities 

, 1 > • . . 1 ■ 1 • 

activities 

1 1 1 1 1 1 l 1 1 ' 

1 1 1 1 1 1 1 1 1 i 1 1 

0 Lacks  intellec-  | Below  average  in  2 Suitable  or  ac-  g Above  average 

tual  curiosity.  finding  means  to  ceptable,  usual-  in  overcoming 

Seldom  starts  overcome  diffi-  ly  goes  ahead  difficulties, 

anything  new  culties  when  started  resourceful 


Starts  new 
undertakings 
habitually  works 
beyond  required 
minimum 


1 1 


1 1 1 


0 Over  sensitive, 

1 Occasionally  im- 

2 Well  poised  most  3 Superior 

self  n 

Exceptional  3 

feelings  easily 

patient,  flies 

of  the  time  control, 

feel- 

poise,  calm- 

hurt  or  excited 

off  the  handle 

ings  not 

easily 

ness  and  good 

hurt 

humor  under 

l , 1 1 1 l ■ 1 , 1 

1 ....  1 ...  . 

1 . ■ . . 1 . . . . 1 . . . i 

1 , , , . 1 , 

stress 

1 1 1 1 1 1 1 1 1 

p Lacks  balance 

| Some  tendency  to 

2 

Acts  judiciously 

3 

Has  habit  of  con-  4 

Inspires  unusual 

5 

and  restraint, 

react  without  re- 

in  ordinary  cir- 

sidered  judgment 

confidence  in 

jumps  at  con- 

straint  or  fail 

cumstances 

thorough  analys is 

soundness  of 

clus ions 

to  forsee  results 

judgment  and 

i 

1 

1 1 1 1 1 1 1 1 1 1 

dec  is  ions 

1 1 . . 1 . 1 . . 

j 

0 Avoids  group  ac- 

| Usually  does  not 

2 

Takes  part  in 

3 

Participates  4 

Seems  happy  in 

5 

tivities,  often 

subordinate  self 

normal  number  of 

frequently  in 

teamwork 

conspicuous  for 

to  group,  diffi- 

group  activities 

teams  and  groups, 

extremely 

poor  teamwork 

cult  to  handle 

meets  others 

always  helpful 

cooperat ive 

1 1 l 1 1 1 1 1 1 1 

1 

half  way 

1 

....  1 ...  1 1 

1 1 1 1 1 1 1 1 1 

1 

0 Keeps  people  at 

■ Does  not  make 

2 

Many  likeable 

3 

Has  many  friends  4 

Has  a great  many 

5 

a distance 

friends  easily 

qualities, 

promotes  good 

friends  unusual- 

others  which 

feeling 

ly  attractive 

repel 

soc  ially 

RELIABILITY 


SELF- 

CONFIDENCE 


0 Neglects  pro-  . 

mises  and  ob- 
ligations 

1 . 1 . 1 1 . . 1 . 1 

Erratic,  may  « 

disappoint  even 
after  being  im- 
pressed with  need 
for  dependability 

Generally  ful- 
fills promises 
and  obligations 

. 1 1 . 1 1 . 1 . 

3 Superior  ful-  4 

fillment  of 
obligat ions 
and  promises 

1 1 . . . 1 . 1 1 1 l 

Dependable  in  5 
word  and  deed 
at  all  times 

iiiii.ii.  1 

0 Exceedingly  timid  . 

Frequently  em-  « 

Moderately  con- 

3 Wholesomely  self-  4 

Excellent  self-  3 

and  hesitant, 

harassed  or  L 

fident  of  him- 

confident,  usual- 

assurance,  al- 

easily  influenced 

flustered 

self 

ly  at  ease 

ways  at  ease 
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